Businesses like banks that provide service have to worry about the problem of 'Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on the improvement of service, keeping in mind these priorities.
Given a Bank customer, build a neural network-based classifier that can determine whether they will leave or not in the next 6 months.
The case study is from an open-source dataset from Kaggle. The dataset contains 10,000 sample points with 14 distinct features such as CustomerId, CreditScore, Geography, Gender, Age, Tenure, Balance, etc.
* 0=No ( Customer did not leave the bank )
* 1=Yes ( Customer left the bank )
import tensorflow as tf
print(tf.__version__)
2.8.0
import pandas as pd
import numpy as np
import seaborn as sns
import sklearn
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
import matplotlib.pyplot as plt
churn_data = pd.read_csv('churn.csv')
churn_data
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 15634602 | Hargrave | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | 2 | 15647311 | Hill | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | 3 | 15619304 | Onio | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | 4 | 15701354 | Boni | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | 5 | 15737888 | Mitchell | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 9995 | 9996 | 15606229 | Obijiaku | 771 | France | Male | 39 | 5 | 0.00 | 2 | 1 | 0 | 96270.64 | 0 |
| 9996 | 9997 | 15569892 | Johnstone | 516 | France | Male | 35 | 10 | 57369.61 | 1 | 1 | 1 | 101699.77 | 0 |
| 9997 | 9998 | 15584532 | Liu | 709 | France | Female | 36 | 7 | 0.00 | 1 | 0 | 1 | 42085.58 | 1 |
| 9998 | 9999 | 15682355 | Sabbatini | 772 | Germany | Male | 42 | 3 | 75075.31 | 2 | 1 | 0 | 92888.52 | 1 |
| 9999 | 10000 | 15628319 | Walker | 792 | France | Female | 28 | 4 | 130142.79 | 1 | 1 | 0 | 38190.78 | 0 |
10000 rows × 14 columns
churn_data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null object 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null object 5 Gender 10000 non-null object 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: float64(2), int64(9), object(3) memory usage: 1.1+ MB
# Let's check for duplicate values in the data
churn_data.duplicated().sum()
0
# Let's check for missing values in the data
round(churn_data.isnull().sum() / churn_data.isnull().count() * 100, 2)
RowNumber 0.0 CustomerId 0.0 Surname 0.0 CreditScore 0.0 Geography 0.0 Gender 0.0 Age 0.0 Tenure 0.0 Balance 0.0 NumOfProducts 0.0 HasCrCard 0.0 IsActiveMember 0.0 EstimatedSalary 0.0 Exited 0.0 dtype: float64
churn_data.describe().T
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| RowNumber | 10000.0 | 5.000500e+03 | 2886.895680 | 1.00 | 2500.75 | 5.000500e+03 | 7.500250e+03 | 10000.00 |
| CustomerId | 10000.0 | 1.569094e+07 | 71936.186123 | 15565701.00 | 15628528.25 | 1.569074e+07 | 1.575323e+07 | 15815690.00 |
| CreditScore | 10000.0 | 6.505288e+02 | 96.653299 | 350.00 | 584.00 | 6.520000e+02 | 7.180000e+02 | 850.00 |
| Age | 10000.0 | 3.892180e+01 | 10.487806 | 18.00 | 32.00 | 3.700000e+01 | 4.400000e+01 | 92.00 |
| Tenure | 10000.0 | 5.012800e+00 | 2.892174 | 0.00 | 3.00 | 5.000000e+00 | 7.000000e+00 | 10.00 |
| Balance | 10000.0 | 7.648589e+04 | 62397.405202 | 0.00 | 0.00 | 9.719854e+04 | 1.276442e+05 | 250898.09 |
| NumOfProducts | 10000.0 | 1.530200e+00 | 0.581654 | 1.00 | 1.00 | 1.000000e+00 | 2.000000e+00 | 4.00 |
| HasCrCard | 10000.0 | 7.055000e-01 | 0.455840 | 0.00 | 0.00 | 1.000000e+00 | 1.000000e+00 | 1.00 |
| IsActiveMember | 10000.0 | 5.151000e-01 | 0.499797 | 0.00 | 0.00 | 1.000000e+00 | 1.000000e+00 | 1.00 |
| EstimatedSalary | 10000.0 | 1.000902e+05 | 57510.492818 | 11.58 | 51002.11 | 1.001939e+05 | 1.493882e+05 | 199992.48 |
| Exited | 10000.0 | 2.037000e-01 | 0.402769 | 0.00 | 0.00 | 0.000000e+00 | 0.000000e+00 | 1.00 |
churn_data.shape
(10000, 14)
churn_data.isna().sum()
RowNumber 0 CustomerId 0 Surname 0 CreditScore 0 Geography 0 Gender 0 Age 0 Tenure 0 Balance 0 NumOfProducts 0 HasCrCard 0 IsActiveMember 0 EstimatedSalary 0 Exited 0 dtype: int64
churn_data.nunique()
RowNumber 10000 CustomerId 10000 Surname 2932 CreditScore 460 Geography 3 Gender 2 Age 70 Tenure 11 Balance 6382 NumOfProducts 4 HasCrCard 2 IsActiveMember 2 EstimatedSalary 9999 Exited 2 dtype: int64
for i in churn_data.describe(include=["object"]).columns:
print("Unique values in", i, "are :")
print(churn_data[i].value_counts())
print("*" * 50)
Unique values in Surname are :
Smith 32
Scott 29
Martin 29
Walker 28
Brown 26
..
Izmailov 1
Bold 1
Bonham 1
Poninski 1
Burbidge 1
Name: Surname, Length: 2932, dtype: int64
**************************************************
Unique values in Geography are :
France 5014
Germany 2509
Spain 2477
Name: Geography, dtype: int64
**************************************************
Unique values in Gender are :
Male 5457
Female 4543
Name: Gender, dtype: int64
**************************************************
df=churn_data.copy()
df=df.drop(["RowNumber","Surname"],axis=1)
churn_data.nunique()
RowNumber 10000 CustomerId 10000 Surname 2932 CreditScore 460 Geography 3 Gender 2 Age 70 Tenure 11 Balance 6382 NumOfProducts 4 HasCrCard 2 IsActiveMember 2 EstimatedSalary 9999 Exited 2 dtype: int64
df.shape
(10000, 12)
# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
"""
Boxplot and histogram combined
data: dataframe
feature: dataframe column
figsize: size of figure (default (12,7))
kde: whether to show the density curve (default False)
bins: number of bins for histogram (default None)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(
nrows=2, # Number of rows of the subplot grid= 2
sharex=True, # x-axis will be shared among all subplots
gridspec_kw={"height_ratios": (0.25, 0.75)},
figsize=figsize,
) # creating the 2 subplots
sns.boxplot(
data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
) # boxplot will be created and a star will indicate the mean value of the column
sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
) if bins else sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2
) # For histogram
ax_hist2.axvline(
data[feature].mean(), color="green", linestyle="--"
) # Add mean to the histogram
ax_hist2.axvline(
data[feature].median(), color="black", linestyle="-"
) # Add median to the histogram
df.columns
Index(['CustomerId', 'CreditScore', 'Geography', 'Gender', 'Age', 'Tenure',
'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember',
'EstimatedSalary', 'Exited'],
dtype='object')
histogram_boxplot(df, "CreditScore")
histogram_boxplot(df, "Age")
histogram_boxplot(df, "Tenure")
histogram_boxplot(df, "Balance")
histogram_boxplot(df, "NumOfProducts")
histogram_boxplot(df, "HasCrCard")
histogram_boxplot(df, "EstimatedSalary")
histogram_boxplot(df, "Exited")
print(df.Exited.value_counts())
labels = 'Churned', 'Not Churned'
#sizes = [ds.is_promoted[ds['is_promoted']==1].count(), ds.is_promoted[ds['is_promoted']==0].count()]
sizes = [df.Exited[df['Exited']==1].count(),df.Exited[df['Exited']==0].count()]
explode = (0, 0.1)
fig1, ax1 = plt.subplots(figsize=(10, 8))
ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',
shadow=True, startangle=90)
ax1.axis('equal')
plt.title("Churn", size = 20)
plt.show()
0 7963 1 2037 Name: Exited, dtype: int64
df.columns
Index(['CustomerId', 'CreditScore', 'Geography', 'Gender', 'Age', 'Tenure',
'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember',
'EstimatedSalary', 'Exited'],
dtype='object')
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
target_uniq = data[target].unique()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[1, 0].set_title("Boxplot w.r.t target")
sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
)
plt.tight_layout()
plt.show()
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
distribution_plot_wrt_target(df, "Age", "Exited")
distribution_plot_wrt_target(df, "CreditScore", "Exited")
# function to plot stacked bar chart
def stacked_barplot(data, predictor, target):
"""
Print the category counts and plot a stacked bar chart
data: dataframe
predictor: independent variable
target: target variable
"""
count = data[predictor].nunique()
sorter = data[target].value_counts().index[-1]
tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
by=sorter, ascending=False
)
print(tab1)
print("-" * 120)
tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
by=sorter, ascending=False
)
tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
plt.legend(
loc="lower left",
frameon=False,
)
plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
plt.show()
stacked_barplot(df, "Gender", "Exited")
Exited 0 1 All Gender All 7963 2037 10000 Female 3404 1139 4543 Male 4559 898 5457 ------------------------------------------------------------------------------------------------------------------------
stacked_barplot(df, "Tenure", "Exited")
Exited 0 1 All Tenure All 7963 2037 10000 1 803 232 1035 3 796 213 1009 9 771 213 984 5 803 209 1012 4 786 203 989 2 847 201 1048 8 828 197 1025 6 771 196 967 7 851 177 1028 10 389 101 490 0 318 95 413 ------------------------------------------------------------------------------------------------------------------------
stacked_barplot(df, "Geography", "Exited")
Exited 0 1 All Geography All 7963 2037 10000 Germany 1695 814 2509 France 4204 810 5014 Spain 2064 413 2477 ------------------------------------------------------------------------------------------------------------------------
stacked_barplot(df, "Age", "Exited")
Exited 0 1 All Age All 7963 2037 10000 46 135 91 226 40 343 89 432 43 209 88 297 45 142 87 229 .. ... ... ... 79 4 0 4 78 5 0 5 77 10 0 10 76 11 0 11 75 9 0 9 [71 rows x 3 columns] ------------------------------------------------------------------------------------------------------------------------
df.nunique()
CustomerId 10000 CreditScore 460 Geography 3 Gender 2 Age 70 Tenure 11 Balance 6382 NumOfProducts 4 HasCrCard 2 IsActiveMember 2 EstimatedSalary 9999 Exited 2 dtype: int64
# for column in df.columns:
labeled_barplot(df, "Age", perc=True)
df.columns
Index(['CustomerId', 'CreditScore', 'Geography', 'Gender', 'Age', 'Tenure',
'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember',
'EstimatedSalary', 'Exited'],
dtype='object')
labeled_barplot(df, "NumOfProducts", perc=True)
labeled_barplot(df, "Tenure", perc=True)
labeled_barplot(df, "IsActiveMember", perc=True)
labeled_barplot(df, "HasCrCard", perc=True)
labeled_barplot(df, "Exited", perc=True)
labeled_barplot(df, "Geography", perc=True)
labeled_barplot(df, "Gender", perc=True)
plt.figure(figsize=(15, 7))
sns.heatmap(
df.corr(), annot=True, vmin=-1, vmax=1, fmt=".3f", cmap="Spectral"
)
plt.show()
sns.pairplot(df,hue = 'Exited',diag_kind = "kde",kind = "scatter",palette = "husl",height=5.5)
plt.show()
-Key meaningful observations from Bivariate analysis
df['CustomerId'] = df['CustomerId'].astype('float32')
df['CreditScore'] = df['CreditScore'].astype('float32')
df['Age'] = df['Age'].astype('float32')
df['Tenure'] = df['Tenure'].astype('float32')
df['NumOfProducts'] = df['NumOfProducts'].astype('float32')
df['HasCrCard'] = df['HasCrCard'].astype('float32')
df['IsActiveMember'] = df['IsActiveMember'].astype('float32')
df['Exited'] = df['Exited'].astype('float32')
df['Geography'] = df['Geography'].astype('category')
df['Gender'] = df['Gender'].astype('category')
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CustomerId 10000 non-null float32 1 CreditScore 10000 non-null float32 2 Geography 10000 non-null category 3 Gender 10000 non-null category 4 Age 10000 non-null float32 5 Tenure 10000 non-null float32 6 Balance 10000 non-null float64 7 NumOfProducts 10000 non-null float32 8 HasCrCard 10000 non-null float32 9 IsActiveMember 10000 non-null float32 10 EstimatedSalary 10000 non-null float64 11 Exited 10000 non-null float32 dtypes: category(2), float32(8), float64(2) memory usage: 488.7 KB
churn_process_df = pd.get_dummies(df,drop_first=True)
churn_process_df = churn_process_df.astype('float32')
churn_process_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 13 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CustomerId 10000 non-null float32 1 CreditScore 10000 non-null float32 2 Age 10000 non-null float32 3 Tenure 10000 non-null float32 4 Balance 10000 non-null float32 5 NumOfProducts 10000 non-null float32 6 HasCrCard 10000 non-null float32 7 IsActiveMember 10000 non-null float32 8 EstimatedSalary 10000 non-null float32 9 Exited 10000 non-null float32 10 Geography_Germany 10000 non-null float32 11 Geography_Spain 10000 non-null float32 12 Gender_Male 10000 non-null float32 dtypes: float32(13) memory usage: 507.9 KB
Y = churn_process_df['Exited']
# churn_model_df=df.copy()
X=churn_process_df.drop("Exited",axis=1)
#Splitting the training and test set
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.20, random_state=1)
#Splitting the train set into dev set and training set
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.20, random_state=1)
# Checking the shape
print(X_train.shape)
print(X_val.shape)
print(y_train.shape)
print(y_val.shape)
(6400, 12) (1600, 12) (6400,) (1600,)
scaler = preprocessing.MinMaxScaler()
# MinMaxScalar has been used here. You can go ahead and use the other scalars available and chcek the effect on the results.
#fitting the transform on test and train separately
X_train = scaler.fit_transform(X_train)
X_test = scaler.fit_transform(X_test)
X_val = scaler.fit_transform(X_val)
# create target scaler object
target_scaler = preprocessing.MinMaxScaler()
y_train = target_scaler.fit_transform(y_train.values.reshape(-1,1))
y_test = target_scaler.fit_transform(y_test.values.reshape(-1,1))
y_val=target_scaler.fit_transform(y_val.values.reshape(-1,1))
#Printing the data after normalization
X_val
array([[0.22304535, 1. , 0.12162164, ..., 0. , 0. ,
1. ],
[0.05264282, 0.564 , 0.43243244, ..., 1. , 0. ,
0. ],
[0.5979347 , 0.17799997, 0.24324325, ..., 0. , 0. ,
1. ],
...,
[0.5916748 , 0.15600002, 0.5945946 , ..., 0. , 0. ,
0. ],
[0.4633026 , 0.658 , 0.13513514, ..., 0. , 1. ,
0. ],
[0.5828018 , 0.65400004, 0.27027026, ..., 0. , 0. ,
1. ]], dtype=float32)
# let us now convert the data elements into tensors as we need tensors to be fed into different tensorflow based operations
#X-train and X_test were converted to numpy arrays while transformations while the other two need to be transformed into numpy arrays.
X_train_scale=tf.convert_to_tensor(X_train)
y_train_scale=tf.convert_to_tensor(y_train)
X_test_scale=tf.convert_to_tensor(X_test)
y_test_scale=tf.convert_to_tensor(y_test)
X_val_scale=tf.convert_to_tensor(X_val)
y_val_scale=tf.convert_to_tensor(y_val)
#printing the shape of training tensor
X_train_scale.shape[1]
12
Predicting customer is going to churn and while customer is not looking to churn
Both are important for us to minimize, the bank would want the F1 Score evaluation metric to be maximized/ Hence, the focus should be on increasing the F1 score rather than focusing on just one metric i.e. Recall or Precision.
import numpy as np
import pandas as pd
from tensorflow.keras.utils import to_categorical
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
from matplotlib import pyplot
from scipy import stats
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import explained_variance_score, mean_squared_error, r2_score, mean_absolute_error
from collections import Counter
from keras import callbacks
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout,InputLayer
from tensorflow.keras.models import Sequential
from keras.layers import BatchNormalization
from tensorflow.keras import regularizers
from tensorflow.keras import backend
from random import shuffle
from keras.callbacks import ModelCheckpoint
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.optimizers import RMSprop
import warnings
warnings.filterwarnings("ignore")
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
# Initializing the ANN
model = Sequential()
# The amount of nodes (dimensions) in hidden layer should be the average of input and output layers, in this case 64.
# This adds the input layer (by specifying input dimension) AND the first hidden layer (units)
model.add(Dense(64,activation = 'relu',input_shape=(X_train_scale.shape[1],)))
#Add 1st hidden layer
model.add(Dense(32, activation='relu'))
# Adding the output layer
# Notice that we do not need to specify input dim.
# we have an output of 1 node, which is the the desired dimensions of our output (stay with the bank or not)
# We use the sigmoid because we want probability outcomes
model.add(Dense(1, activation = 'sigmoid'))
# Create optimizer with default learning rate
# Compile the model
model.compile(optimizer='SGD', loss='binary_crossentropy', metrics=['accuracy'])
history=model.fit(X_train_scale, y_train_scale,
validation_split=0.2,
epochs=100,
batch_size=32,verbose=1)
Epoch 1/100 160/160 [==============================] - 1s 2ms/step - loss: 0.5375 - accuracy: 0.7904 - val_loss: 0.5083 - val_accuracy: 0.7906 Epoch 2/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4924 - accuracy: 0.8000 - val_loss: 0.5023 - val_accuracy: 0.7906 Epoch 3/100 160/160 [==============================] - 0s 991us/step - loss: 0.4869 - accuracy: 0.8000 - val_loss: 0.4978 - val_accuracy: 0.7906 Epoch 4/100 160/160 [==============================] - 0s 978us/step - loss: 0.4825 - accuracy: 0.8000 - val_loss: 0.4930 - val_accuracy: 0.7906 Epoch 5/100 160/160 [==============================] - 0s 885us/step - loss: 0.4786 - accuracy: 0.8000 - val_loss: 0.4896 - val_accuracy: 0.7906 Epoch 6/100 160/160 [==============================] - 0s 918us/step - loss: 0.4754 - accuracy: 0.8000 - val_loss: 0.4860 - val_accuracy: 0.7906 Epoch 7/100 160/160 [==============================] - 0s 971us/step - loss: 0.4724 - accuracy: 0.8000 - val_loss: 0.4833 - val_accuracy: 0.7906 Epoch 8/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4700 - accuracy: 0.8000 - val_loss: 0.4809 - val_accuracy: 0.7906 Epoch 9/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4676 - accuracy: 0.8000 - val_loss: 0.4781 - val_accuracy: 0.7906 Epoch 10/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4657 - accuracy: 0.8000 - val_loss: 0.4759 - val_accuracy: 0.7906 Epoch 11/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4639 - accuracy: 0.8000 - val_loss: 0.4741 - val_accuracy: 0.7906 Epoch 12/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4622 - accuracy: 0.8000 - val_loss: 0.4726 - val_accuracy: 0.7906 Epoch 13/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4608 - accuracy: 0.8000 - val_loss: 0.4708 - val_accuracy: 0.7906 Epoch 14/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4591 - accuracy: 0.8000 - val_loss: 0.4691 - val_accuracy: 0.7906 Epoch 15/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4577 - accuracy: 0.8000 - val_loss: 0.4679 - val_accuracy: 0.7906 Epoch 16/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4564 - accuracy: 0.8000 - val_loss: 0.4662 - val_accuracy: 0.7906 Epoch 17/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4550 - accuracy: 0.7998 - val_loss: 0.4645 - val_accuracy: 0.7906 Epoch 18/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4534 - accuracy: 0.8002 - val_loss: 0.4643 - val_accuracy: 0.7906 Epoch 19/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4522 - accuracy: 0.8006 - val_loss: 0.4617 - val_accuracy: 0.7906 Epoch 20/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4509 - accuracy: 0.8008 - val_loss: 0.4605 - val_accuracy: 0.7930 Epoch 21/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4495 - accuracy: 0.8020 - val_loss: 0.4586 - val_accuracy: 0.7937 Epoch 22/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4481 - accuracy: 0.8041 - val_loss: 0.4583 - val_accuracy: 0.7930 Epoch 23/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4469 - accuracy: 0.8039 - val_loss: 0.4573 - val_accuracy: 0.7937 Epoch 24/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4456 - accuracy: 0.8041 - val_loss: 0.4552 - val_accuracy: 0.7961 Epoch 25/100 160/160 [==============================] - 0s 981us/step - loss: 0.4443 - accuracy: 0.8043 - val_loss: 0.4528 - val_accuracy: 0.7992 Epoch 26/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4431 - accuracy: 0.8053 - val_loss: 0.4517 - val_accuracy: 0.7977 Epoch 27/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4416 - accuracy: 0.8045 - val_loss: 0.4500 - val_accuracy: 0.8023 Epoch 28/100 160/160 [==============================] - 0s 949us/step - loss: 0.4401 - accuracy: 0.8072 - val_loss: 0.4501 - val_accuracy: 0.7977 Epoch 29/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4394 - accuracy: 0.8082 - val_loss: 0.4479 - val_accuracy: 0.7992 Epoch 30/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4377 - accuracy: 0.8082 - val_loss: 0.4472 - val_accuracy: 0.8000 Epoch 31/100 160/160 [==============================] - 0s 956us/step - loss: 0.4363 - accuracy: 0.8102 - val_loss: 0.4465 - val_accuracy: 0.8016 Epoch 32/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4356 - accuracy: 0.8086 - val_loss: 0.4437 - val_accuracy: 0.8094 Epoch 33/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4345 - accuracy: 0.8111 - val_loss: 0.4435 - val_accuracy: 0.8062 Epoch 34/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4335 - accuracy: 0.8121 - val_loss: 0.4424 - val_accuracy: 0.8094 Epoch 35/100 160/160 [==============================] - 0s 955us/step - loss: 0.4324 - accuracy: 0.8107 - val_loss: 0.4408 - val_accuracy: 0.8141 Epoch 36/100 160/160 [==============================] - 0s 980us/step - loss: 0.4312 - accuracy: 0.8104 - val_loss: 0.4398 - val_accuracy: 0.8156 Epoch 37/100 160/160 [==============================] - 0s 972us/step - loss: 0.4304 - accuracy: 0.8129 - val_loss: 0.4395 - val_accuracy: 0.8148 Epoch 38/100 160/160 [==============================] - 0s 879us/step - loss: 0.4294 - accuracy: 0.8119 - val_loss: 0.4392 - val_accuracy: 0.8141 Epoch 39/100 160/160 [==============================] - 0s 864us/step - loss: 0.4289 - accuracy: 0.8139 - val_loss: 0.4383 - val_accuracy: 0.8148 Epoch 40/100 160/160 [==============================] - 0s 814us/step - loss: 0.4279 - accuracy: 0.8145 - val_loss: 0.4391 - val_accuracy: 0.8109 Epoch 41/100 160/160 [==============================] - 0s 862us/step - loss: 0.4273 - accuracy: 0.8135 - val_loss: 0.4381 - val_accuracy: 0.8102 Epoch 42/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4267 - accuracy: 0.8141 - val_loss: 0.4378 - val_accuracy: 0.8109 Epoch 43/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4261 - accuracy: 0.8156 - val_loss: 0.4359 - val_accuracy: 0.8156 Epoch 44/100 160/160 [==============================] - 0s 975us/step - loss: 0.4257 - accuracy: 0.8158 - val_loss: 0.4352 - val_accuracy: 0.8164 Epoch 45/100 160/160 [==============================] - 0s 864us/step - loss: 0.4248 - accuracy: 0.8154 - val_loss: 0.4362 - val_accuracy: 0.8156 Epoch 46/100 160/160 [==============================] - 0s 971us/step - loss: 0.4243 - accuracy: 0.8158 - val_loss: 0.4342 - val_accuracy: 0.8148 Epoch 47/100 160/160 [==============================] - 0s 898us/step - loss: 0.4239 - accuracy: 0.8172 - val_loss: 0.4338 - val_accuracy: 0.8172 Epoch 48/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4232 - accuracy: 0.8180 - val_loss: 0.4349 - val_accuracy: 0.8164 Epoch 49/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4224 - accuracy: 0.8164 - val_loss: 0.4337 - val_accuracy: 0.8141 Epoch 50/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4226 - accuracy: 0.8164 - val_loss: 0.4329 - val_accuracy: 0.8180 Epoch 51/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4221 - accuracy: 0.8174 - val_loss: 0.4338 - val_accuracy: 0.8172 Epoch 52/100 160/160 [==============================] - 0s 989us/step - loss: 0.4216 - accuracy: 0.8180 - val_loss: 0.4331 - val_accuracy: 0.8172 Epoch 53/100 160/160 [==============================] - 0s 873us/step - loss: 0.4212 - accuracy: 0.8164 - val_loss: 0.4324 - val_accuracy: 0.8180 Epoch 54/100 160/160 [==============================] - 0s 958us/step - loss: 0.4205 - accuracy: 0.8162 - val_loss: 0.4328 - val_accuracy: 0.8180 Epoch 55/100 160/160 [==============================] - 0s 942us/step - loss: 0.4202 - accuracy: 0.8170 - val_loss: 0.4317 - val_accuracy: 0.8188 Epoch 56/100 160/160 [==============================] - 0s 940us/step - loss: 0.4198 - accuracy: 0.8180 - val_loss: 0.4325 - val_accuracy: 0.8172 Epoch 57/100 160/160 [==============================] - 0s 985us/step - loss: 0.4196 - accuracy: 0.8172 - val_loss: 0.4316 - val_accuracy: 0.8219 Epoch 58/100 160/160 [==============================] - 0s 946us/step - loss: 0.4187 - accuracy: 0.8180 - val_loss: 0.4310 - val_accuracy: 0.8164 Epoch 59/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4186 - accuracy: 0.8180 - val_loss: 0.4308 - val_accuracy: 0.8172 Epoch 60/100 160/160 [==============================] - 0s 847us/step - loss: 0.4182 - accuracy: 0.8164 - val_loss: 0.4320 - val_accuracy: 0.8188 Epoch 61/100 160/160 [==============================] - 0s 839us/step - loss: 0.4177 - accuracy: 0.8207 - val_loss: 0.4320 - val_accuracy: 0.8164 Epoch 62/100 160/160 [==============================] - 0s 903us/step - loss: 0.4174 - accuracy: 0.8195 - val_loss: 0.4300 - val_accuracy: 0.8188 Epoch 63/100 160/160 [==============================] - 0s 899us/step - loss: 0.4161 - accuracy: 0.8193 - val_loss: 0.4302 - val_accuracy: 0.8211 Epoch 64/100 160/160 [==============================] - 0s 962us/step - loss: 0.4162 - accuracy: 0.8207 - val_loss: 0.4293 - val_accuracy: 0.8203 Epoch 65/100 160/160 [==============================] - 0s 836us/step - loss: 0.4154 - accuracy: 0.8223 - val_loss: 0.4291 - val_accuracy: 0.8227 Epoch 66/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4151 - accuracy: 0.8209 - val_loss: 0.4289 - val_accuracy: 0.8203 Epoch 67/100 160/160 [==============================] - 0s 860us/step - loss: 0.4148 - accuracy: 0.8217 - val_loss: 0.4297 - val_accuracy: 0.8242 Epoch 68/100 160/160 [==============================] - 0s 873us/step - loss: 0.4143 - accuracy: 0.8223 - val_loss: 0.4295 - val_accuracy: 0.8242 Epoch 69/100 160/160 [==============================] - 0s 924us/step - loss: 0.4132 - accuracy: 0.8242 - val_loss: 0.4312 - val_accuracy: 0.8203 Epoch 70/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4132 - accuracy: 0.8244 - val_loss: 0.4276 - val_accuracy: 0.8227 Epoch 71/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4125 - accuracy: 0.8258 - val_loss: 0.4269 - val_accuracy: 0.8250 Epoch 72/100 160/160 [==============================] - 0s 875us/step - loss: 0.4117 - accuracy: 0.8266 - val_loss: 0.4265 - val_accuracy: 0.8250 Epoch 73/100 160/160 [==============================] - 0s 906us/step - loss: 0.4109 - accuracy: 0.8254 - val_loss: 0.4261 - val_accuracy: 0.8273 Epoch 74/100 160/160 [==============================] - 0s 897us/step - loss: 0.4105 - accuracy: 0.8273 - val_loss: 0.4261 - val_accuracy: 0.8250 Epoch 75/100 160/160 [==============================] - 0s 837us/step - loss: 0.4096 - accuracy: 0.8293 - val_loss: 0.4278 - val_accuracy: 0.8242 Epoch 76/100 160/160 [==============================] - 0s 921us/step - loss: 0.4090 - accuracy: 0.8293 - val_loss: 0.4256 - val_accuracy: 0.8289 Epoch 77/100 160/160 [==============================] - 0s 1ms/step - loss: 0.4084 - accuracy: 0.8285 - val_loss: 0.4245 - val_accuracy: 0.8289 Epoch 78/100 160/160 [==============================] - 0s 941us/step - loss: 0.4077 - accuracy: 0.8283 - val_loss: 0.4236 - val_accuracy: 0.8281 Epoch 79/100 160/160 [==============================] - 0s 871us/step - loss: 0.4068 - accuracy: 0.8314 - val_loss: 0.4230 - val_accuracy: 0.8305 Epoch 80/100 160/160 [==============================] - 0s 973us/step - loss: 0.4061 - accuracy: 0.8309 - val_loss: 0.4242 - val_accuracy: 0.8266 Epoch 81/100 160/160 [==============================] - 0s 942us/step - loss: 0.4051 - accuracy: 0.8322 - val_loss: 0.4214 - val_accuracy: 0.8297 Epoch 82/100 160/160 [==============================] - 0s 877us/step - loss: 0.4043 - accuracy: 0.8322 - val_loss: 0.4220 - val_accuracy: 0.8297 Epoch 83/100 160/160 [==============================] - 0s 933us/step - loss: 0.4036 - accuracy: 0.8326 - val_loss: 0.4198 - val_accuracy: 0.8320 Epoch 84/100 160/160 [==============================] - 0s 861us/step - loss: 0.4024 - accuracy: 0.8320 - val_loss: 0.4210 - val_accuracy: 0.8289 Epoch 85/100 160/160 [==============================] - 0s 878us/step - loss: 0.4015 - accuracy: 0.8338 - val_loss: 0.4186 - val_accuracy: 0.8313 Epoch 86/100 160/160 [==============================] - 0s 887us/step - loss: 0.4004 - accuracy: 0.8338 - val_loss: 0.4171 - val_accuracy: 0.8328 Epoch 87/100 160/160 [==============================] - 0s 960us/step - loss: 0.3995 - accuracy: 0.8334 - val_loss: 0.4164 - val_accuracy: 0.8344 Epoch 88/100 160/160 [==============================] - 0s 964us/step - loss: 0.3988 - accuracy: 0.8326 - val_loss: 0.4167 - val_accuracy: 0.8320 Epoch 89/100 160/160 [==============================] - 0s 949us/step - loss: 0.3979 - accuracy: 0.8342 - val_loss: 0.4145 - val_accuracy: 0.8313 Epoch 90/100 160/160 [==============================] - 0s 974us/step - loss: 0.3967 - accuracy: 0.8375 - val_loss: 0.4136 - val_accuracy: 0.8336 Epoch 91/100 160/160 [==============================] - 0s 880us/step - loss: 0.3954 - accuracy: 0.8371 - val_loss: 0.4185 - val_accuracy: 0.8305 Epoch 92/100 160/160 [==============================] - 0s 887us/step - loss: 0.3944 - accuracy: 0.8367 - val_loss: 0.4115 - val_accuracy: 0.8352 Epoch 93/100 160/160 [==============================] - 0s 897us/step - loss: 0.3936 - accuracy: 0.8359 - val_loss: 0.4106 - val_accuracy: 0.8352 Epoch 94/100 160/160 [==============================] - 0s 945us/step - loss: 0.3927 - accuracy: 0.8393 - val_loss: 0.4094 - val_accuracy: 0.8352 Epoch 95/100 160/160 [==============================] - 0s 1ms/step - loss: 0.3914 - accuracy: 0.8369 - val_loss: 0.4085 - val_accuracy: 0.8359 Epoch 96/100 160/160 [==============================] - 0s 1ms/step - loss: 0.3902 - accuracy: 0.8402 - val_loss: 0.4074 - val_accuracy: 0.8375 Epoch 97/100 160/160 [==============================] - 0s 992us/step - loss: 0.3891 - accuracy: 0.8391 - val_loss: 0.4081 - val_accuracy: 0.8344 Epoch 98/100 160/160 [==============================] - 0s 867us/step - loss: 0.3874 - accuracy: 0.8406 - val_loss: 0.4056 - val_accuracy: 0.8367 Epoch 99/100 160/160 [==============================] - 0s 841us/step - loss: 0.3862 - accuracy: 0.8408 - val_loss: 0.4100 - val_accuracy: 0.8383 Epoch 100/100 160/160 [==============================] - 0s 875us/step - loss: 0.3857 - accuracy: 0.8428 - val_loss: 0.4031 - val_accuracy: 0.8398
# Capturing learning history per epoch
hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
# Plotting accuracy at different epochs
plt.plot(hist['loss'])
plt.plot(hist['val_loss'])
plt.legend(("train" , "valid") , loc =0)
#Printing results
results = model.evaluate(X_test_scale, y_test_scale)
63/63 [==============================] - 0s 1ms/step - loss: 0.3891 - accuracy: 0.8375
y_pred_scale=model.predict(X_test_scale)
y_pred_scale = (y_pred_scale > 0.5)
y_pred_scale
array([[False],
[False],
[False],
...,
[False],
[False],
[False]])
def make_confusion_matrix(cf,
group_names=None,
categories='auto',
count=True,
percent=True,
cbar=True,
xyticks=True,
xyplotlabels=True,
sum_stats=True,
figsize=None,
cmap='Blues',
title=None):
'''
This function will make a pretty plot of an sklearn Confusion Matrix cm using a Seaborn heatmap visualization.
Arguments
'''
# CODE TO GENERATE TEXT INSIDE EACH SQUARE
blanks = ['' for i in range(cf.size)]
if group_names and len(group_names)==cf.size:
group_labels = ["{}\n".format(value) for value in group_names]
else:
group_labels = blanks
if count:
group_counts = ["{0:0.0f}\n".format(value) for value in cf.flatten()]
else:
group_counts = blanks
if percent:
group_percentages = ["{0:.2%}".format(value) for value in cf.flatten()/np.sum(cf)]
else:
group_percentages = blanks
box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
box_labels = np.asarray(box_labels).reshape(cf.shape[0],cf.shape[1])
# CODE TO GENERATE SUMMARY STATISTICS & TEXT FOR SUMMARY STATS
if sum_stats:
#Accuracy is sum of diagonal divided by total observations
accuracy = np.trace(cf) / float(np.sum(cf))
# SET FIGURE PARAMETERS ACCORDING TO OTHER ARGUMENTS
if figsize==None:
#Get default figure size if not set
figsize = plt.rcParams.get('figure.figsize')
if xyticks==False:
#Do not show categories if xyticks is False
categories=False
# MAKE THE HEATMAP VISUALIZATION
plt.figure(figsize=figsize)
sns.heatmap(cf,annot=box_labels,fmt="",cmap=cmap,cbar=cbar,xticklabels=categories,yticklabels=categories)
if title:
plt.title(title)
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm=confusion_matrix(y_test_scale, y_pred_scale)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr=metrics.classification_report(y_test_scale,y_pred_scale)
print(cr)
precision recall f1-score support
0.0 0.85 0.96 0.90 1585
1.0 0.72 0.35 0.47 415
accuracy 0.84 2000
macro avg 0.79 0.66 0.69 2000
weighted avg 0.82 0.84 0.81 2000
1) Imbalanced dataset: As you have seen in the EDA, this dataset is imbalanced, and it contains more examples that belong to the 0 class.
2) Decision Threshold: Due to the imbalanced dataset, we can use ROC-AUC to find the optimal threshold and use the same for prediction.
Let's try to change the optimizer, tune the decision threshold, increase the layers and configure some other hyperparameters accordingly, in order to improve the model's performance.
-Comment on which metric is right for model performance evaluation and why? - Find the optimal threshold using ROC-AUC curves - Comment on model performance - Can model performance be improved? check and comment - Build another model to implement these improvements - Include all the model which were trained to reach at the final one
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
model1 = Sequential()
#Adding the hidden and output layers
model1.add(Dense(256,activation='relu',kernel_initializer='he_uniform',input_dim = X_train_scale.shape[1]))
model1.add(Dense(128,activation='relu',kernel_initializer='he_uniform'))
model1.add(Dense(64,activation='relu',kernel_initializer='he_uniform'))
model1.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
model1.add(Dense(1, activation = 'sigmoid'))
#Compiling the ANN with Adam optimizer and binary cross entropy loss function
optimizer = tf.keras.optimizers.Adam(0.001)
model1.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
model1.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 256) 3328
dense_1 (Dense) (None, 128) 32896
dense_2 (Dense) (None, 64) 8256
dense_3 (Dense) (None, 32) 2080
dense_4 (Dense) (None, 1) 33
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________
history1 = model1.fit(X_train_scale,y_train_scale,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)
Epoch 1/50 80/80 [==============================] - 1s 2ms/step - loss: 0.4775 - accuracy: 0.7953 - val_loss: 0.4749 - val_accuracy: 0.7922 Epoch 2/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4424 - accuracy: 0.8135 - val_loss: 0.4359 - val_accuracy: 0.8297 Epoch 3/50 80/80 [==============================] - 0s 1ms/step - loss: 0.4069 - accuracy: 0.8295 - val_loss: 0.4292 - val_accuracy: 0.8203 Epoch 4/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3747 - accuracy: 0.8469 - val_loss: 0.3940 - val_accuracy: 0.8422 Epoch 5/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3535 - accuracy: 0.8547 - val_loss: 0.3732 - val_accuracy: 0.8391 Epoch 6/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3541 - accuracy: 0.8521 - val_loss: 0.3737 - val_accuracy: 0.8414 Epoch 7/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3364 - accuracy: 0.8586 - val_loss: 0.3691 - val_accuracy: 0.8492 Epoch 8/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3321 - accuracy: 0.8625 - val_loss: 0.3805 - val_accuracy: 0.8398 Epoch 9/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3292 - accuracy: 0.8615 - val_loss: 0.3649 - val_accuracy: 0.8453 Epoch 10/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3225 - accuracy: 0.8658 - val_loss: 0.4037 - val_accuracy: 0.8391 Epoch 11/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3218 - accuracy: 0.8666 - val_loss: 0.3649 - val_accuracy: 0.8469 Epoch 12/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3159 - accuracy: 0.8654 - val_loss: 0.3731 - val_accuracy: 0.8531 Epoch 13/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3108 - accuracy: 0.8721 - val_loss: 0.3700 - val_accuracy: 0.8367 Epoch 14/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3059 - accuracy: 0.8723 - val_loss: 0.3728 - val_accuracy: 0.8531 Epoch 15/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2964 - accuracy: 0.8754 - val_loss: 0.3746 - val_accuracy: 0.8492 Epoch 16/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2939 - accuracy: 0.8773 - val_loss: 0.3755 - val_accuracy: 0.8484 Epoch 17/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2948 - accuracy: 0.8764 - val_loss: 0.3780 - val_accuracy: 0.8438 Epoch 18/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2827 - accuracy: 0.8826 - val_loss: 0.4061 - val_accuracy: 0.8477 Epoch 19/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2815 - accuracy: 0.8811 - val_loss: 0.3864 - val_accuracy: 0.8508 Epoch 20/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2706 - accuracy: 0.8902 - val_loss: 0.3992 - val_accuracy: 0.8359 Epoch 21/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2719 - accuracy: 0.8891 - val_loss: 0.3996 - val_accuracy: 0.8523 Epoch 22/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2671 - accuracy: 0.8896 - val_loss: 0.3965 - val_accuracy: 0.8477 Epoch 23/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2601 - accuracy: 0.8906 - val_loss: 0.4130 - val_accuracy: 0.8398 Epoch 24/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2565 - accuracy: 0.8918 - val_loss: 0.4213 - val_accuracy: 0.8469 Epoch 25/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2533 - accuracy: 0.8949 - val_loss: 0.4119 - val_accuracy: 0.8297 Epoch 26/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2442 - accuracy: 0.8990 - val_loss: 0.4165 - val_accuracy: 0.8297 Epoch 27/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2451 - accuracy: 0.8984 - val_loss: 0.4197 - val_accuracy: 0.8414 Epoch 28/50 80/80 [==============================] - 0s 1ms/step - loss: 0.2370 - accuracy: 0.9033 - val_loss: 0.4409 - val_accuracy: 0.8414 Epoch 29/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2358 - accuracy: 0.9029 - val_loss: 0.4318 - val_accuracy: 0.8430 Epoch 30/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2284 - accuracy: 0.9070 - val_loss: 0.4573 - val_accuracy: 0.8391 Epoch 31/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2217 - accuracy: 0.9082 - val_loss: 0.4575 - val_accuracy: 0.8398 Epoch 32/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2113 - accuracy: 0.9152 - val_loss: 0.4797 - val_accuracy: 0.8414 Epoch 33/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2122 - accuracy: 0.9182 - val_loss: 0.4857 - val_accuracy: 0.8406 Epoch 34/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2070 - accuracy: 0.9176 - val_loss: 0.4931 - val_accuracy: 0.8430 Epoch 35/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2026 - accuracy: 0.9178 - val_loss: 0.5099 - val_accuracy: 0.8391 Epoch 36/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1972 - accuracy: 0.9205 - val_loss: 0.5053 - val_accuracy: 0.8406 Epoch 37/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1976 - accuracy: 0.9197 - val_loss: 0.5166 - val_accuracy: 0.8156 Epoch 38/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1938 - accuracy: 0.9203 - val_loss: 0.5246 - val_accuracy: 0.8344 Epoch 39/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1834 - accuracy: 0.9225 - val_loss: 0.5415 - val_accuracy: 0.8328 Epoch 40/50 80/80 [==============================] - 0s 1ms/step - loss: 0.1765 - accuracy: 0.9305 - val_loss: 0.5524 - val_accuracy: 0.8281 Epoch 41/50 80/80 [==============================] - 0s 1ms/step - loss: 0.1669 - accuracy: 0.9330 - val_loss: 0.5779 - val_accuracy: 0.8406 Epoch 42/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1740 - accuracy: 0.9301 - val_loss: 0.5669 - val_accuracy: 0.8438 Epoch 43/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1609 - accuracy: 0.9342 - val_loss: 0.5679 - val_accuracy: 0.8234 Epoch 44/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1589 - accuracy: 0.9344 - val_loss: 0.6220 - val_accuracy: 0.8250 Epoch 45/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1566 - accuracy: 0.9359 - val_loss: 0.6145 - val_accuracy: 0.8250 Epoch 46/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1492 - accuracy: 0.9406 - val_loss: 0.6397 - val_accuracy: 0.8188 Epoch 47/50 80/80 [==============================] - 0s 1ms/step - loss: 0.1406 - accuracy: 0.9449 - val_loss: 0.6310 - val_accuracy: 0.8227 Epoch 48/50 80/80 [==============================] - 0s 1ms/step - loss: 0.1379 - accuracy: 0.9447 - val_loss: 0.6821 - val_accuracy: 0.8148 Epoch 49/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1428 - accuracy: 0.9404 - val_loss: 0.6652 - val_accuracy: 0.8195 Epoch 50/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1391 - accuracy: 0.9434 - val_loss: 0.6665 - val_accuracy: 0.8125
#Plotting Train Loss vs Validation Loss
plt.plot(history1.history['loss'])
plt.plot(history1.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
from sklearn.metrics import roc_curve
from matplotlib import pyplot
# predict probabilities
yhat1 = model1.predict(X_test_scale)
# keep probabilities for the positive outcome only
yhat1 = yhat1[:, 0]
# calculate roc curves
fpr, tpr, thresholds1 = roc_curve(y_test_scale, yhat1)
# calculate the g-mean for each threshold
gmeans1 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans1)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds1[ix], gmeans1[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='NO Churn')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()
Best Threshold=0.085891, G-Mean=0.730
#Predicting the results using best as a threshold
y_pred_e1=model1.predict(X_test_scale)
y_pred_e1 = (y_pred_e1 > thresholds1[ix])
y_pred_e1
array([[False],
[False],
[ True],
...,
[False],
[False],
[ True]])
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm1=confusion_matrix(y_test_scale, y_pred_e1)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm1,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr1=metrics.classification_report(y_test_scale,y_pred_e1)
print(cr1)
precision recall f1-score support
0.0 0.91 0.75 0.82 1585
1.0 0.42 0.71 0.53 415
accuracy 0.74 2000
macro avg 0.67 0.73 0.68 2000
weighted avg 0.81 0.74 0.76 2000
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
model2 = Sequential()
model2.add(Dense(128,activation='relu',input_dim = X_train_scale.shape[1]))
model2.add(BatchNormalization())
model2.add(Dense(64,activation='relu',kernel_initializer='he_uniform'))
model2.add(BatchNormalization())
model2.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
model2.add(Dense(1, activation = 'sigmoid'))
model2.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 128) 1664
batch_normalization (BatchN (None, 128) 512
ormalization)
dense_1 (Dense) (None, 64) 8256
batch_normalization_1 (Batc (None, 64) 256
hNormalization)
dense_2 (Dense) (None, 32) 2080
dense_3 (Dense) (None, 1) 33
=================================================================
Total params: 12,801
Trainable params: 12,417
Non-trainable params: 384
_________________________________________________________________
optimizer = tf.keras.optimizers.Adam(0.001)
model2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_2 = model2.fit(X_train_scale,y_train_scale,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)
Epoch 1/50 80/80 [==============================] - 1s 3ms/step - loss: 0.5102 - accuracy: 0.7555 - val_loss: 0.5450 - val_accuracy: 0.7906 Epoch 2/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4003 - accuracy: 0.8330 - val_loss: 0.4895 - val_accuracy: 0.7906 Epoch 3/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3662 - accuracy: 0.8480 - val_loss: 0.4553 - val_accuracy: 0.7937 Epoch 4/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3447 - accuracy: 0.8562 - val_loss: 0.4186 - val_accuracy: 0.8117 Epoch 5/50 80/80 [==============================] - 0s 1ms/step - loss: 0.3331 - accuracy: 0.8648 - val_loss: 0.4145 - val_accuracy: 0.8180 Epoch 6/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3308 - accuracy: 0.8609 - val_loss: 0.3883 - val_accuracy: 0.8422 Epoch 7/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3163 - accuracy: 0.8686 - val_loss: 0.3900 - val_accuracy: 0.8445 Epoch 8/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3125 - accuracy: 0.8652 - val_loss: 0.3821 - val_accuracy: 0.8555 Epoch 9/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3064 - accuracy: 0.8703 - val_loss: 0.3782 - val_accuracy: 0.8594 Epoch 10/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3058 - accuracy: 0.8686 - val_loss: 0.3788 - val_accuracy: 0.8367 Epoch 11/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3006 - accuracy: 0.8740 - val_loss: 0.3817 - val_accuracy: 0.8484 Epoch 12/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2893 - accuracy: 0.8801 - val_loss: 0.3886 - val_accuracy: 0.8508 Epoch 13/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2952 - accuracy: 0.8713 - val_loss: 0.3875 - val_accuracy: 0.8477 Epoch 14/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2880 - accuracy: 0.8783 - val_loss: 0.3935 - val_accuracy: 0.8398 Epoch 15/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2828 - accuracy: 0.8797 - val_loss: 0.4020 - val_accuracy: 0.8516 Epoch 16/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2818 - accuracy: 0.8809 - val_loss: 0.4016 - val_accuracy: 0.8414 Epoch 17/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2754 - accuracy: 0.8885 - val_loss: 0.4026 - val_accuracy: 0.8469 Epoch 18/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2769 - accuracy: 0.8809 - val_loss: 0.4024 - val_accuracy: 0.8492 Epoch 19/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2675 - accuracy: 0.8893 - val_loss: 0.3993 - val_accuracy: 0.8492 Epoch 20/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2624 - accuracy: 0.8902 - val_loss: 0.4128 - val_accuracy: 0.8477 Epoch 21/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2643 - accuracy: 0.8887 - val_loss: 0.4213 - val_accuracy: 0.8438 Epoch 22/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2598 - accuracy: 0.8943 - val_loss: 0.4255 - val_accuracy: 0.8422 Epoch 23/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2583 - accuracy: 0.8900 - val_loss: 0.4244 - val_accuracy: 0.8430 Epoch 24/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2526 - accuracy: 0.8926 - val_loss: 0.4193 - val_accuracy: 0.8391 Epoch 25/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2447 - accuracy: 0.8975 - val_loss: 0.4259 - val_accuracy: 0.8352 Epoch 26/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2411 - accuracy: 0.8998 - val_loss: 0.4380 - val_accuracy: 0.8375 Epoch 27/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2416 - accuracy: 0.8969 - val_loss: 0.4261 - val_accuracy: 0.8344 Epoch 28/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2386 - accuracy: 0.8975 - val_loss: 0.4288 - val_accuracy: 0.8422 Epoch 29/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2331 - accuracy: 0.9033 - val_loss: 0.4393 - val_accuracy: 0.8406 Epoch 30/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2299 - accuracy: 0.9051 - val_loss: 0.4607 - val_accuracy: 0.8438 Epoch 31/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2283 - accuracy: 0.9014 - val_loss: 0.4442 - val_accuracy: 0.8359 Epoch 32/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2228 - accuracy: 0.9021 - val_loss: 0.4685 - val_accuracy: 0.8320 Epoch 33/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2219 - accuracy: 0.9062 - val_loss: 0.4522 - val_accuracy: 0.8289 Epoch 34/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2178 - accuracy: 0.9100 - val_loss: 0.4701 - val_accuracy: 0.8406 Epoch 35/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2144 - accuracy: 0.9111 - val_loss: 0.4706 - val_accuracy: 0.8313 Epoch 36/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2074 - accuracy: 0.9129 - val_loss: 0.4833 - val_accuracy: 0.8313 Epoch 37/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2163 - accuracy: 0.9080 - val_loss: 0.4659 - val_accuracy: 0.8297 Epoch 38/50 80/80 [==============================] - 0s 1ms/step - loss: 0.2075 - accuracy: 0.9094 - val_loss: 0.4796 - val_accuracy: 0.8250 Epoch 39/50 80/80 [==============================] - 0s 1ms/step - loss: 0.2038 - accuracy: 0.9137 - val_loss: 0.4804 - val_accuracy: 0.8391 Epoch 40/50 80/80 [==============================] - 0s 1ms/step - loss: 0.2027 - accuracy: 0.9121 - val_loss: 0.4846 - val_accuracy: 0.8328 Epoch 41/50 80/80 [==============================] - 0s 1ms/step - loss: 0.1972 - accuracy: 0.9197 - val_loss: 0.4964 - val_accuracy: 0.8359 Epoch 42/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1941 - accuracy: 0.9182 - val_loss: 0.5089 - val_accuracy: 0.8219 Epoch 43/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1918 - accuracy: 0.9213 - val_loss: 0.5067 - val_accuracy: 0.8344 Epoch 44/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1879 - accuracy: 0.9242 - val_loss: 0.5357 - val_accuracy: 0.8289 Epoch 45/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1885 - accuracy: 0.9225 - val_loss: 0.5205 - val_accuracy: 0.8336 Epoch 46/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1853 - accuracy: 0.9234 - val_loss: 0.5238 - val_accuracy: 0.8250 Epoch 47/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1761 - accuracy: 0.9242 - val_loss: 0.5027 - val_accuracy: 0.8211 Epoch 48/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1753 - accuracy: 0.9268 - val_loss: 0.5400 - val_accuracy: 0.8273 Epoch 49/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1781 - accuracy: 0.9242 - val_loss: 0.5225 - val_accuracy: 0.8289 Epoch 50/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1725 - accuracy: 0.9240 - val_loss: 0.5405 - val_accuracy: 0.8273
#Plotting Train Loss vs Validation Loss
plt.plot(history_2.history['loss'])
plt.plot(history_2.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
from sklearn.metrics import roc_curve
from matplotlib import pyplot
# predict probabilities
yhat2 = model2.predict(X_test_scale)
# keep probabilities for the positive outcome only
yhat2 = yhat2[:, 0]
# calculate roc curves
fpr, tpr, thresholds2 = roc_curve(y_test_scale, yhat2)
# calculate the g-mean for each threshold
gmeans2 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans2)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds2[ix], gmeans2[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()
Best Threshold=0.174853, G-Mean=0.741
y_pred_e2=model2.predict(X_test_scale)
y_pred_e2 = (y_pred_e2 > thresholds2[ix])
y_pred_e2
array([[False],
[ True],
[False],
...,
[False],
[False],
[ True]])
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm2=confusion_matrix(y_test_scale, y_pred_e2)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr3=metrics.classification_report(y_test_scale,y_pred_e2)
print(cr3)
precision recall f1-score support
0.0 0.91 0.78 0.84 1585
1.0 0.46 0.70 0.55 415
accuracy 0.77 2000
macro avg 0.68 0.74 0.70 2000
weighted avg 0.82 0.77 0.78 2000
The Train and Validation curves seem to show overfitting despite having a good F1 score and a low False Negative rate.
Let's try to use the Dropout technique and check to see if it can reduce the False Negative rate.
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
model3 = Sequential()
model3.add(Dense(256,activation='relu',input_dim = X_train_scale.shape[1]))
model3.add(Dropout(0.2))
model3.add(Dense(128,activation='relu'))
model3.add(Dropout(0.2))
model3.add(Dense(64,activation='relu'))
model3.add(Dropout(0.2))
model3.add(Dense(32,activation='relu'))
model3.add(Dense(1, activation = 'sigmoid'))
model3.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 256) 3328
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 128) 32896
dropout_1 (Dropout) (None, 128) 0
dense_2 (Dense) (None, 64) 8256
dropout_2 (Dropout) (None, 64) 0
dense_3 (Dense) (None, 32) 2080
dense_4 (Dense) (None, 1) 33
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________
optimizer = tf.keras.optimizers.Adam(0.001)
model3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_3 = model3.fit(X_train_scale,y_train_scale,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)
Epoch 1/50 80/80 [==============================] - 1s 3ms/step - loss: 0.4967 - accuracy: 0.7922 - val_loss: 0.4701 - val_accuracy: 0.7906 Epoch 2/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4594 - accuracy: 0.8002 - val_loss: 0.4551 - val_accuracy: 0.7906 Epoch 3/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4447 - accuracy: 0.8043 - val_loss: 0.4337 - val_accuracy: 0.8188 Epoch 4/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4203 - accuracy: 0.8270 - val_loss: 0.4161 - val_accuracy: 0.8328 Epoch 5/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3962 - accuracy: 0.8328 - val_loss: 0.3929 - val_accuracy: 0.8406 Epoch 6/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3847 - accuracy: 0.8412 - val_loss: 0.3844 - val_accuracy: 0.8430 Epoch 7/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3713 - accuracy: 0.8486 - val_loss: 0.3743 - val_accuracy: 0.8469 Epoch 8/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3621 - accuracy: 0.8510 - val_loss: 0.3699 - val_accuracy: 0.8484 Epoch 9/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3548 - accuracy: 0.8535 - val_loss: 0.3661 - val_accuracy: 0.8430 Epoch 10/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3544 - accuracy: 0.8529 - val_loss: 0.3679 - val_accuracy: 0.8422 Epoch 11/50 80/80 [==============================] - 0s 1ms/step - loss: 0.3474 - accuracy: 0.8592 - val_loss: 0.3599 - val_accuracy: 0.8547 Epoch 12/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3496 - accuracy: 0.8562 - val_loss: 0.3650 - val_accuracy: 0.8539 Epoch 13/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3407 - accuracy: 0.8637 - val_loss: 0.3629 - val_accuracy: 0.8531 Epoch 14/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3353 - accuracy: 0.8639 - val_loss: 0.3665 - val_accuracy: 0.8539 Epoch 15/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3339 - accuracy: 0.8629 - val_loss: 0.3561 - val_accuracy: 0.8570 Epoch 16/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3283 - accuracy: 0.8670 - val_loss: 0.3550 - val_accuracy: 0.8586 Epoch 17/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3323 - accuracy: 0.8631 - val_loss: 0.3603 - val_accuracy: 0.8516 Epoch 18/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3259 - accuracy: 0.8676 - val_loss: 0.3795 - val_accuracy: 0.8570 Epoch 19/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3235 - accuracy: 0.8715 - val_loss: 0.3644 - val_accuracy: 0.8562 Epoch 20/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3169 - accuracy: 0.8719 - val_loss: 0.3667 - val_accuracy: 0.8516 Epoch 21/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3218 - accuracy: 0.8658 - val_loss: 0.3698 - val_accuracy: 0.8562 Epoch 22/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3207 - accuracy: 0.8691 - val_loss: 0.3613 - val_accuracy: 0.8586 Epoch 23/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3153 - accuracy: 0.8652 - val_loss: 0.3694 - val_accuracy: 0.8539 Epoch 24/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3205 - accuracy: 0.8682 - val_loss: 0.3811 - val_accuracy: 0.8523 Epoch 25/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3129 - accuracy: 0.8699 - val_loss: 0.3651 - val_accuracy: 0.8578 Epoch 26/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3090 - accuracy: 0.8729 - val_loss: 0.3630 - val_accuracy: 0.8531 Epoch 27/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3124 - accuracy: 0.8723 - val_loss: 0.3737 - val_accuracy: 0.8516 Epoch 28/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3051 - accuracy: 0.8742 - val_loss: 0.3738 - val_accuracy: 0.8508 Epoch 29/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3043 - accuracy: 0.8803 - val_loss: 0.3692 - val_accuracy: 0.8477 Epoch 30/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3085 - accuracy: 0.8734 - val_loss: 0.3682 - val_accuracy: 0.8508 Epoch 31/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3079 - accuracy: 0.8721 - val_loss: 0.3684 - val_accuracy: 0.8578 Epoch 32/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2988 - accuracy: 0.8783 - val_loss: 0.3685 - val_accuracy: 0.8523 Epoch 33/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3016 - accuracy: 0.8748 - val_loss: 0.3696 - val_accuracy: 0.8578 Epoch 34/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3026 - accuracy: 0.8750 - val_loss: 0.3793 - val_accuracy: 0.8570 Epoch 35/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2935 - accuracy: 0.8834 - val_loss: 0.3767 - val_accuracy: 0.8539 Epoch 36/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2942 - accuracy: 0.8775 - val_loss: 0.3826 - val_accuracy: 0.8500 Epoch 37/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2887 - accuracy: 0.8814 - val_loss: 0.3794 - val_accuracy: 0.8398 Epoch 38/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2870 - accuracy: 0.8836 - val_loss: 0.3915 - val_accuracy: 0.8547 Epoch 39/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2985 - accuracy: 0.8730 - val_loss: 0.3918 - val_accuracy: 0.8484 Epoch 40/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2905 - accuracy: 0.8824 - val_loss: 0.3794 - val_accuracy: 0.8453 Epoch 41/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2848 - accuracy: 0.8869 - val_loss: 0.3898 - val_accuracy: 0.8477 Epoch 42/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2863 - accuracy: 0.8834 - val_loss: 0.3919 - val_accuracy: 0.8438 Epoch 43/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2781 - accuracy: 0.8852 - val_loss: 0.3972 - val_accuracy: 0.8500 Epoch 44/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2800 - accuracy: 0.8863 - val_loss: 0.3922 - val_accuracy: 0.8414 Epoch 45/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2823 - accuracy: 0.8854 - val_loss: 0.3898 - val_accuracy: 0.8461 Epoch 46/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2778 - accuracy: 0.8846 - val_loss: 0.4007 - val_accuracy: 0.8438 Epoch 47/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2741 - accuracy: 0.8869 - val_loss: 0.3875 - val_accuracy: 0.8500 Epoch 48/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2731 - accuracy: 0.8883 - val_loss: 0.3946 - val_accuracy: 0.8539 Epoch 49/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2733 - accuracy: 0.8867 - val_loss: 0.3920 - val_accuracy: 0.8516 Epoch 50/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2783 - accuracy: 0.8857 - val_loss: 0.3987 - val_accuracy: 0.8469
#Plotting Train Loss vs Validation Loss
plt.plot(history_3.history['loss'])
plt.plot(history_3.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
from sklearn.metrics import roc_curve
from matplotlib import pyplot
# predict probabilities
yhat3 = model3.predict(X_test_scale)
# keep probabilities for the positive outcome only
yhat3 = yhat3[:, 0]
# calculate roc curves
fpr, tpr, thresholds3 = roc_curve(y_test_scale, yhat3)
# calculate the g-mean for each threshold
gmeans3 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans3)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds3[ix], gmeans3[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Churning')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()
Best Threshold=0.223874, G-Mean=0.775
y_pred_e3=model3.predict(X_test)
y_pred_e3 = (y_pred_e3 > thresholds3[ix])
y_pred_e3
array([[False],
[False],
[False],
...,
[False],
[False],
[ True]])
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm3=confusion_matrix(y_test_scale, y_pred_e3)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm3,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr3=metrics.classification_report(y_test_scale,y_pred_e3)
print(cr3)
precision recall f1-score support
0.0 0.92 0.83 0.87 1585
1.0 0.52 0.73 0.61 415
accuracy 0.80 2000
macro avg 0.72 0.78 0.74 2000
weighted avg 0.84 0.80 0.82 2000
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
def create_model_v4(lr,batch_size):
np.random.seed(1337)
model = Sequential()
model.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
model.add(Dropout(0.3))
#model.add(Dense(128,activation='relu',kernel_initializer='he_uniform'))
model.add(Dense(128,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(64,activation='relu'))
model.add(Dropout(0.2))
#model.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
#model.add(Dropout(0.3))
model.add(Dense(32,activation='relu'))
model.add(Dense(1, activation='sigmoid'))
#compile model
optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
model.compile(optimizer = optimizer,loss = 'binary_crossentropy', metrics = ['accuracy'])
return model
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
keras_estimator = KerasClassifier(build_fn=create_model_v4, verbose=1)
# define the grid search parameters
param_random = {
'batch_size':[32, 64, 128],
"lr":[0.01,0.1,0.001],}
kfold_splits = 3
random= RandomizedSearchCV(estimator=keras_estimator,
verbose=1,
cv=kfold_splits,
param_distributions=param_random,n_jobs=-1)
X_train_scale
<tf.Tensor: shape=(6400, 12), dtype=float32, numpy=
array([[0.70726776, 0.32799995, 0.22972974, ..., 0. , 0. ,
1. ],
[0.5607605 , 0.804 , 0.24324325, ..., 0. , 1. ,
1. ],
[0.7751503 , 0.58800006, 0.29729733, ..., 0. , 0. ,
0. ],
...,
[0.55343246, 0.72 , 0.5945946 , ..., 0. , 1. ,
1. ],
[0.86558914, 0.6700001 , 0.2027027 , ..., 0. , 0. ,
1. ],
[0.42694092, 0.48800004, 0.47297296, ..., 0. , 0. ,
0. ]], dtype=float32)>
random_result = random.fit(X_train, y_train,validation_split=0.2,verbose=1)
# Summarize results
print("Best: %f using %s" % (random_result.best_score_, random_result.best_params_))
means = random_result.cv_results_['mean_test_score']
stds = random_result.cv_results_['std_test_score']
params = random_result.cv_results_['params']
Fitting 3 folds for each of 9 candidates, totalling 27 fits
107/107 [==============================] - 2s 9ms/step - loss: 0.4883 - accuracy: 0.7981 - val_loss: 0.5066 - val_accuracy: 0.7775
107/107 [==============================] - 2s 9ms/step - loss: 0.4919 - accuracy: 0.8011 - val_loss: 0.4556 - val_accuracy: 0.8162
107/107 [==============================] - 2s 9ms/step - loss: 0.4975 - accuracy: 0.7940 - val_loss: 0.5071 - val_accuracy: 0.7775
107/107 [==============================] - 2s 9ms/step - loss: 1.1301 - accuracy: 0.7931 - val_loss: 0.5252 - val_accuracy: 0.7775
107/107 [==============================] - 2s 9ms/step - loss: 1.1195 - accuracy: 0.7902 - val_loss: 0.4778 - val_accuracy: 0.8162
107/107 [==============================] - 2s 8ms/step - loss: 0.7842 - accuracy: 0.7861 - val_loss: 0.5301 - val_accuracy: 0.7775
54/54 [==============================] - 2s 10ms/step - loss: 0.4867 - accuracy: 0.7931 - val_loss: 0.4951 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.4859 - accuracy: 0.8045 - val_loss: 0.4982 - val_accuracy: 0.7775
107/107 [==============================] - 2s 7ms/step - loss: 0.5045 - accuracy: 0.7887 - val_loss: 0.4941 - val_accuracy: 0.7775
54/54 [==============================] - 2s 9ms/step - loss: 0.5174 - accuracy: 0.7887 - val_loss: 0.4556 - val_accuracy: 0.8162
54/54 [==============================] - 2s 9ms/step - loss: 0.4950 - accuracy: 0.7882 - val_loss: 0.4865 - val_accuracy: 0.7775
107/107 [==============================] - 2s 7ms/step - loss: 0.4953 - accuracy: 0.7984 - val_loss: 0.4556 - val_accuracy: 0.8162
67/67 [==============================] - 0s 2ms/step - loss: 0.4735 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4742 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.5020 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4793 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4610 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.5260 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.4867 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.4646 - accuracy: 0.7957
34/34 [==============================] - 0s 2ms/step - loss: 0.4578 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.4591 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4748 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.4764 - accuracy: 0.7857
27/27 [==============================] - 2s 25ms/step - loss: 0.4855 - accuracy: 0.7937 - val_loss: 0.4856 - val_accuracy: 0.7775
54/54 [==============================] - 2s 14ms/step - loss: 1.9249 - accuracy: 0.7879 - val_loss: 0.4782 - val_accuracy: 0.8162
54/54 [==============================] - 2s 14ms/step - loss: 1.6557 - accuracy: 0.7975 - val_loss: 0.5376 - val_accuracy: 0.7775
27/27 [==============================] - 2s 22ms/step - loss: 0.4904 - accuracy: 0.7841 - val_loss: 0.5239 - val_accuracy: 0.7775
54/54 [==============================] - 2s 14ms/step - loss: 1.3663 - accuracy: 0.7823 - val_loss: 0.5327 - val_accuracy: 0.7775
27/27 [==============================] - 2s 18ms/step - loss: 2.6068 - accuracy: 0.7860 - val_loss: 0.5263 - val_accuracy: 0.7775
27/27 [==============================] - 2s 18ms/step - loss: 0.5068 - accuracy: 0.7879 - val_loss: 0.4697 - val_accuracy: 0.8162
54/54 [==============================] - 2s 13ms/step - loss: 0.5126 - accuracy: 0.7887 - val_loss: 0.5025 - val_accuracy: 0.7775
27/27 [==============================] - 2s 19ms/step - loss: 1.9176 - accuracy: 0.7627 - val_loss: 0.5328 - val_accuracy: 0.7775
54/54 [==============================] - 2s 13ms/step - loss: 0.5112 - accuracy: 0.7998 - val_loss: 0.5050 - val_accuracy: 0.7775
54/54 [==============================] - 2s 12ms/step - loss: 0.5365 - accuracy: 0.7770 - val_loss: 0.4601 - val_accuracy: 0.8162
27/27 [==============================] - 2s 14ms/step - loss: 2.7694 - accuracy: 0.7829 - val_loss: 0.4767 - val_accuracy: 0.8162
17/17 [==============================] - 0s 2ms/step - loss: 0.4639 - accuracy: 0.7957
17/17 [==============================] - 0s 2ms/step - loss: 0.4716 - accuracy: 0.8129
17/17 [==============================] - 0s 3ms/step - loss: 0.5000 - accuracy: 0.7957
17/17 [==============================] - 0s 2ms/step - loss: 0.4834 - accuracy: 0.7857
17/17 [==============================] - 0s 3ms/step - loss: 0.4820 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.5093 - accuracy: 0.7957
34/34 [==============================] - 0s 2ms/step - loss: 0.5200 - accuracy: 0.7857
34/34 [==============================] - 0s 3ms/step - loss: 0.4819 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4652 - accuracy: 0.8129
34/34 [==============================] - 0s 3ms/step - loss: 0.4835 - accuracy: 0.7957
34/34 [==============================] - 0s 3ms/step - loss: 0.4936 - accuracy: 0.7857
17/17 [==============================] - 0s 4ms/step - loss: 0.5241 - accuracy: 0.7857
27/27 [==============================] - 1s 10ms/step - loss: 0.5499 - accuracy: 0.7734 - val_loss: 0.5147 - val_accuracy: 0.7775
27/27 [==============================] - 1s 10ms/step - loss: 0.5519 - accuracy: 0.7653 - val_loss: 0.5118 - val_accuracy: 0.7775
27/27 [==============================] - 1s 9ms/step - loss: 0.5584 - accuracy: 0.7732 - val_loss: 0.4660 - val_accuracy: 0.8162
17/17 [==============================] - 0s 980us/step - loss: 0.4912 - accuracy: 0.7957
17/17 [==============================] - 0s 980us/step - loss: 0.4702 - accuracy: 0.8129
17/17 [==============================] - 0s 821us/step - loss: 0.5037 - accuracy: 0.7857
160/160 [==============================] - 1s 2ms/step - loss: 0.4794 - accuracy: 0.7982 - val_loss: 0.5614 - val_accuracy: 0.7906
Best: 0.798125 using {'lr': 0.01, 'batch_size': 32}
estimator_v4=create_model_v4(batch_size=random_result.best_params_['batch_size'],lr=random_result.best_params_['lr'])
estimator_v4.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_5 (Dense) (None, 256) 3328
dropout_3 (Dropout) (None, 256) 0
dense_6 (Dense) (None, 128) 32896
dropout_4 (Dropout) (None, 128) 0
dense_7 (Dense) (None, 64) 8256
dropout_5 (Dropout) (None, 64) 0
dense_8 (Dense) (None, 32) 2080
dense_9 (Dense) (None, 1) 33
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________
optimizer = tf.keras.optimizers.Adam(random_result.best_params_['lr'])
estimator_v4.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_4=estimator_v4.fit(X_train, y_train, epochs=50, batch_size = 64, verbose=1,validation_split=0.2)
Epoch 1/50 80/80 [==============================] - 1s 3ms/step - loss: 0.4759 - accuracy: 0.7986 - val_loss: 0.4963 - val_accuracy: 0.7906 Epoch 2/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4559 - accuracy: 0.8025 - val_loss: 0.4282 - val_accuracy: 0.8250 Epoch 3/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4251 - accuracy: 0.8180 - val_loss: 0.4176 - val_accuracy: 0.8398 Epoch 4/50 80/80 [==============================] - 0s 1ms/step - loss: 0.4090 - accuracy: 0.8268 - val_loss: 0.4047 - val_accuracy: 0.8328 Epoch 5/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3763 - accuracy: 0.8463 - val_loss: 0.3881 - val_accuracy: 0.8406 Epoch 6/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3883 - accuracy: 0.8328 - val_loss: 0.3742 - val_accuracy: 0.8461 Epoch 7/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3818 - accuracy: 0.8373 - val_loss: 0.3668 - val_accuracy: 0.8430 Epoch 8/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3766 - accuracy: 0.8436 - val_loss: 0.3773 - val_accuracy: 0.8508 Epoch 9/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3706 - accuracy: 0.8465 - val_loss: 0.3648 - val_accuracy: 0.8445 Epoch 10/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3719 - accuracy: 0.8473 - val_loss: 0.3741 - val_accuracy: 0.8531 Epoch 11/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3735 - accuracy: 0.8508 - val_loss: 0.3788 - val_accuracy: 0.8516 Epoch 12/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3685 - accuracy: 0.8432 - val_loss: 0.3832 - val_accuracy: 0.8516 Epoch 13/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3681 - accuracy: 0.8486 - val_loss: 0.3705 - val_accuracy: 0.8531 Epoch 14/50 80/80 [==============================] - 0s 1ms/step - loss: 0.3635 - accuracy: 0.8549 - val_loss: 0.3763 - val_accuracy: 0.8516 Epoch 15/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3609 - accuracy: 0.8518 - val_loss: 0.3628 - val_accuracy: 0.8570 Epoch 16/50 80/80 [==============================] - 0s 1ms/step - loss: 0.3591 - accuracy: 0.8553 - val_loss: 0.3775 - val_accuracy: 0.8500 Epoch 17/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8502 - val_loss: 0.3658 - val_accuracy: 0.8547 Epoch 18/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3616 - accuracy: 0.8562 - val_loss: 0.3664 - val_accuracy: 0.8547 Epoch 19/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8516 - val_loss: 0.3710 - val_accuracy: 0.8539 Epoch 20/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3579 - accuracy: 0.8537 - val_loss: 0.3619 - val_accuracy: 0.8453 Epoch 21/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3634 - accuracy: 0.8508 - val_loss: 0.3781 - val_accuracy: 0.8602 Epoch 22/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3612 - accuracy: 0.8529 - val_loss: 0.3656 - val_accuracy: 0.8547 Epoch 23/50 80/80 [==============================] - 0s 3ms/step - loss: 0.3569 - accuracy: 0.8545 - val_loss: 0.3692 - val_accuracy: 0.8531 Epoch 24/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8568 - val_loss: 0.3752 - val_accuracy: 0.8531 Epoch 25/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8512 - val_loss: 0.3632 - val_accuracy: 0.8469 Epoch 26/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3588 - accuracy: 0.8506 - val_loss: 0.3640 - val_accuracy: 0.8469 Epoch 27/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3621 - accuracy: 0.8518 - val_loss: 0.3699 - val_accuracy: 0.8547 Epoch 28/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3605 - accuracy: 0.8533 - val_loss: 0.3709 - val_accuracy: 0.8508 Epoch 29/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3598 - accuracy: 0.8523 - val_loss: 0.3677 - val_accuracy: 0.8539 Epoch 30/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3502 - accuracy: 0.8547 - val_loss: 0.3634 - val_accuracy: 0.8516 Epoch 31/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3530 - accuracy: 0.8504 - val_loss: 0.3743 - val_accuracy: 0.8562 Epoch 32/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3511 - accuracy: 0.8535 - val_loss: 0.3610 - val_accuracy: 0.8531 Epoch 33/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3483 - accuracy: 0.8541 - val_loss: 0.3994 - val_accuracy: 0.8562 Epoch 34/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3538 - accuracy: 0.8559 - val_loss: 0.3675 - val_accuracy: 0.8539 Epoch 35/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3602 - accuracy: 0.8541 - val_loss: 0.3685 - val_accuracy: 0.8516 Epoch 36/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3594 - accuracy: 0.8500 - val_loss: 0.3665 - val_accuracy: 0.8516 Epoch 37/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3517 - accuracy: 0.8574 - val_loss: 0.3620 - val_accuracy: 0.8555 Epoch 38/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3516 - accuracy: 0.8539 - val_loss: 0.3639 - val_accuracy: 0.8555 Epoch 39/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3500 - accuracy: 0.8531 - val_loss: 0.3656 - val_accuracy: 0.8461 Epoch 40/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3564 - accuracy: 0.8551 - val_loss: 0.3642 - val_accuracy: 0.8508 Epoch 41/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3491 - accuracy: 0.8551 - val_loss: 0.3615 - val_accuracy: 0.8547 Epoch 42/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8529 - val_loss: 0.3666 - val_accuracy: 0.8523 Epoch 43/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8516 - val_loss: 0.3716 - val_accuracy: 0.8508 Epoch 44/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3488 - accuracy: 0.8572 - val_loss: 0.3554 - val_accuracy: 0.8578 Epoch 45/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3584 - accuracy: 0.8562 - val_loss: 0.3741 - val_accuracy: 0.8516 Epoch 46/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3482 - accuracy: 0.8592 - val_loss: 0.3619 - val_accuracy: 0.8477 Epoch 47/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3555 - accuracy: 0.8527 - val_loss: 0.3627 - val_accuracy: 0.8469 Epoch 48/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3470 - accuracy: 0.8590 - val_loss: 0.3644 - val_accuracy: 0.8500 Epoch 49/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3484 - accuracy: 0.8578 - val_loss: 0.3621 - val_accuracy: 0.8484 Epoch 50/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3508 - accuracy: 0.8539 - val_loss: 0.3562 - val_accuracy: 0.8547
#Plotting Train Loss vs Validation Loss
plt.plot(history_4.history['loss'])
plt.plot(history_4.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
from sklearn.metrics import roc_curve
from matplotlib import pyplot
# predict probabilities
yhat4 = estimator_v4.predict(X_test)
# keep probabilities for the positive outcome only
yhat4 = yhat4[:, 0]
# calculate roc curves
fpr, tpr, thresholds4 = roc_curve(y_test, yhat4)
# calculate the g-mean for each threshold
gmeans4 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans4)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds4[ix], gmeans4[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()
Best Threshold=0.204202, G-Mean=0.779
y_pred_e4=estimator_v4.predict(X_test)
y_pred_e4 = (y_pred_e4 > thresholds4[ix])
y_pred_e4
array([[False],
[False],
[False],
...,
[False],
[False],
[ True]])
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm4=confusion_matrix(y_test, y_pred_e4)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Changing Job','Changing Job']
make_confusion_matrix(cm4,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr4=metrics.classification_report(y_test,y_pred_e4)
print(cr4)
precision recall f1-score support
0.0 0.93 0.80 0.86 1585
1.0 0.49 0.76 0.60 415
accuracy 0.79 2000
macro avg 0.71 0.78 0.73 2000
weighted avg 0.84 0.79 0.80 2000
Accuracy, f1 score has decreased little however the ROC curve is smooth between train and validation
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
def create_model_v5(lr,batch_size):
np.random.seed(1337)
model = Sequential()
model.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
model.add(Dropout(0.3))
#model.add(Dense(128,activation='relu',kernel_initializer='he_uniform'))
model.add(Dense(128,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(64,activation='relu'))
model.add(Dropout(0.2))
#model.add(Dense(32,activation='relu',kernel_initializer='he_uniform'))
#model.add(Dropout(0.3))
model.add(Dense(32,activation='relu'))
model.add(Dense(1, activation='sigmoid'))
#compile model
optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
model.compile(optimizer = optimizer,loss = 'binary_crossentropy', metrics = ['accuracy'])
return model
keras_estimator = KerasClassifier(build_fn=create_model_v5, verbose=1)
# define the grid search parameters
param_grid = {
'batch_size':[64,32, 128],
"lr":[0.01,0.1,0.001],}
kfold_splits = 3
grid = GridSearchCV(estimator=keras_estimator,
verbose=1,
cv=kfold_splits,
param_grid=param_grid,n_jobs=-1)
import time
# store starting time
begin = time.time()
grid_result = grid.fit(X_train, y_train,validation_split=0.2,verbose=1)
# Summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")
Fitting 3 folds for each of 9 candidates, totalling 27 fits
54/54 [==============================] - 2s 15ms/step - loss: 1.8962 - accuracy: 0.7834 - val_loss: 0.5321 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 0.4962 - accuracy: 0.7926 - val_loss: 0.4664 - val_accuracy: 0.8162
54/54 [==============================] - 2s 16ms/step - loss: 0.4941 - accuracy: 0.7914 - val_loss: 0.4870 - val_accuracy: 0.7775
54/54 [==============================] - 2s 16ms/step - loss: 0.4827 - accuracy: 0.7943 - val_loss: 0.4894 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 1.1364 - accuracy: 0.7846 - val_loss: 0.5308 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 0.5069 - accuracy: 0.8007 - val_loss: 0.4977 - val_accuracy: 0.7775
54/54 [==============================] - 2s 15ms/step - loss: 0.5366 - accuracy: 0.7776 - val_loss: 0.4647 - val_accuracy: 0.8162
54/54 [==============================] - 2s 16ms/step - loss: 2.2594 - accuracy: 0.7984 - val_loss: 0.4787 - val_accuracy: 0.8162
54/54 [==============================] - 2s 15ms/step - loss: 0.5015 - accuracy: 0.7937 - val_loss: 0.5002 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.4833 - accuracy: 0.7923 - val_loss: 0.4773 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.4962 - accuracy: 0.8051 - val_loss: 0.4852 - val_accuracy: 0.7775
34/34 [==============================] - 0s 2ms/step - loss: 0.4583 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4831 - accuracy: 0.8129
34/34 [==============================] - 0s 2ms/step - loss: 0.4956 - accuracy: 0.7857
34/34 [==============================] - 0s 2ms/step - loss: 0.4715 - accuracy: 0.7957
107/107 [==============================] - 2s 7ms/step - loss: 0.4842 - accuracy: 0.7958 - val_loss: 0.4701 - val_accuracy: 0.8162
34/34 [==============================] - 0s 4ms/step - loss: 0.5057 - accuracy: 0.7957
34/34 [==============================] - 0s 4ms/step - loss: 0.4819 - accuracy: 0.7857
34/34 [==============================] - 0s 4ms/step - loss: 0.5197 - accuracy: 0.7857
34/34 [==============================] - 0s 4ms/step - loss: 0.4575 - accuracy: 0.8129
34/34 [==============================] - 0s 5ms/step - loss: 0.4659 - accuracy: 0.7957
67/67 [==============================] - 0s 3ms/step - loss: 0.4631 - accuracy: 0.7957
67/67 [==============================] - 0s 3ms/step - loss: 0.4437 - accuracy: 0.8129
67/67 [==============================] - 0s 3ms/step - loss: 0.4804 - accuracy: 0.7857
27/27 [==============================] - 2s 29ms/step - loss: 0.4866 - accuracy: 0.8025 - val_loss: 0.4877 - val_accuracy: 0.7775
27/27 [==============================] - 2s 28ms/step - loss: 0.5073 - accuracy: 0.7814 - val_loss: 0.4748 - val_accuracy: 0.8162
27/27 [==============================] - 2s 26ms/step - loss: 0.5063 - accuracy: 0.7770 - val_loss: 0.4929 - val_accuracy: 0.7775
27/27 [==============================] - 2s 16ms/step - loss: 3.7837 - accuracy: 0.7987 - val_loss: 0.5368 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.7278 - accuracy: 0.7914 - val_loss: 0.4777 - val_accuracy: 0.8162
107/107 [==============================] - 2s 9ms/step - loss: 0.9144 - accuracy: 0.7966 - val_loss: 0.5311 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.9764 - accuracy: 0.7882 - val_loss: 0.5300 - val_accuracy: 0.7775
17/17 [==============================] - 0s 3ms/step - loss: 0.4671 - accuracy: 0.7957
17/17 [==============================] - 0s 3ms/step - loss: 0.4909 - accuracy: 0.7857
107/107 [==============================] - 2s 9ms/step - loss: 0.5251 - accuracy: 0.7668 - val_loss: 0.4966 - val_accuracy: 0.7775
27/27 [==============================] - 2s 17ms/step - loss: 2.8845 - accuracy: 0.7644 - val_loss: 0.5305 - val_accuracy: 0.7775
17/17 [==============================] - 0s 2ms/step - loss: 0.4579 - accuracy: 0.8129
107/107 [==============================] - 2s 8ms/step - loss: 0.5071 - accuracy: 0.7863 - val_loss: 0.4954 - val_accuracy: 0.7775
107/107 [==============================] - 2s 8ms/step - loss: 0.5058 - accuracy: 0.7908 - val_loss: 0.4599 - val_accuracy: 0.8162
27/27 [==============================] - 2s 16ms/step - loss: 3.4141 - accuracy: 0.7923 - val_loss: 0.4773 - val_accuracy: 0.8162
17/17 [==============================] - 0s 3ms/step - loss: 0.4927 - accuracy: 0.7957
17/17 [==============================] - 0s 3ms/step - loss: 0.4836 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.5205 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.5063 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4854 - accuracy: 0.8129
67/67 [==============================] - 0s 2ms/step - loss: 0.4582 - accuracy: 0.8129
17/17 [==============================] - 0s 2ms/step - loss: 0.5212 - accuracy: 0.7857
67/67 [==============================] - 0s 2ms/step - loss: 0.4685 - accuracy: 0.7957
67/67 [==============================] - 0s 2ms/step - loss: 0.4922 - accuracy: 0.7857
27/27 [==============================] - 1s 8ms/step - loss: 0.5340 - accuracy: 0.7870 - val_loss: 0.4781 - val_accuracy: 0.8162
27/27 [==============================] - 1s 8ms/step - loss: 0.5269 - accuracy: 0.7805 - val_loss: 0.5099 - val_accuracy: 0.7775
27/27 [==============================] - 1s 8ms/step - loss: 0.5165 - accuracy: 0.7940 - val_loss: 0.5054 - val_accuracy: 0.7775
17/17 [==============================] - 0s 1ms/step - loss: 0.5029 - accuracy: 0.7857
17/17 [==============================] - 0s 1ms/step - loss: 0.4845 - accuracy: 0.7957
17/17 [==============================] - 0s 999us/step - loss: 0.4709 - accuracy: 0.8129
80/80 [==============================] - 1s 3ms/step - loss: 0.4741 - accuracy: 0.7969 - val_loss: 0.4620 - val_accuracy: 0.7906
Best: 0.798125 using {'batch_size': 64, 'lr': 0.01}
Total runtime of the program is 7.652716636657715
estimator_v5=create_model_v5(batch_size=grid_result.best_params_['batch_size'],lr=grid_result.best_params_['lr'])
estimator_v5.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_5 (Dense) (None, 256) 3328
dropout_3 (Dropout) (None, 256) 0
dense_6 (Dense) (None, 128) 32896
dropout_4 (Dropout) (None, 128) 0
dense_7 (Dense) (None, 64) 8256
dropout_5 (Dropout) (None, 64) 0
dense_8 (Dense) (None, 32) 2080
dense_9 (Dense) (None, 1) 33
=================================================================
Total params: 46,593
Trainable params: 46,593
Non-trainable params: 0
_________________________________________________________________
optimizer = tf.keras.optimizers.Adam(grid_result.best_params_['lr'])
estimator_v5.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_5=estimator_v5.fit(X_train, y_train, epochs=50, batch_size = 64, verbose=1,validation_split=0.2)
Epoch 1/50 80/80 [==============================] - 1s 3ms/step - loss: 0.4759 - accuracy: 0.7986 - val_loss: 0.4963 - val_accuracy: 0.7906 Epoch 2/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4559 - accuracy: 0.8025 - val_loss: 0.4282 - val_accuracy: 0.8250 Epoch 3/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4251 - accuracy: 0.8180 - val_loss: 0.4176 - val_accuracy: 0.8398 Epoch 4/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4090 - accuracy: 0.8268 - val_loss: 0.4047 - val_accuracy: 0.8328 Epoch 5/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3763 - accuracy: 0.8463 - val_loss: 0.3881 - val_accuracy: 0.8406 Epoch 6/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3883 - accuracy: 0.8328 - val_loss: 0.3742 - val_accuracy: 0.8461 Epoch 7/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3818 - accuracy: 0.8373 - val_loss: 0.3668 - val_accuracy: 0.8430 Epoch 8/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3766 - accuracy: 0.8436 - val_loss: 0.3773 - val_accuracy: 0.8508 Epoch 9/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3706 - accuracy: 0.8465 - val_loss: 0.3648 - val_accuracy: 0.8445 Epoch 10/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3719 - accuracy: 0.8473 - val_loss: 0.3741 - val_accuracy: 0.8531 Epoch 11/50 80/80 [==============================] - 0s 3ms/step - loss: 0.3735 - accuracy: 0.8508 - val_loss: 0.3788 - val_accuracy: 0.8516 Epoch 12/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3685 - accuracy: 0.8432 - val_loss: 0.3832 - val_accuracy: 0.8516 Epoch 13/50 80/80 [==============================] - 0s 1ms/step - loss: 0.3681 - accuracy: 0.8486 - val_loss: 0.3705 - val_accuracy: 0.8531 Epoch 14/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3635 - accuracy: 0.8549 - val_loss: 0.3763 - val_accuracy: 0.8516 Epoch 15/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3609 - accuracy: 0.8518 - val_loss: 0.3628 - val_accuracy: 0.8570 Epoch 16/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3591 - accuracy: 0.8553 - val_loss: 0.3775 - val_accuracy: 0.8500 Epoch 17/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8502 - val_loss: 0.3658 - val_accuracy: 0.8547 Epoch 18/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3616 - accuracy: 0.8562 - val_loss: 0.3664 - val_accuracy: 0.8547 Epoch 19/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3622 - accuracy: 0.8516 - val_loss: 0.3710 - val_accuracy: 0.8539 Epoch 20/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3579 - accuracy: 0.8537 - val_loss: 0.3619 - val_accuracy: 0.8453 Epoch 21/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3634 - accuracy: 0.8508 - val_loss: 0.3781 - val_accuracy: 0.8602 Epoch 22/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3612 - accuracy: 0.8529 - val_loss: 0.3656 - val_accuracy: 0.8547 Epoch 23/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3569 - accuracy: 0.8545 - val_loss: 0.3692 - val_accuracy: 0.8531 Epoch 24/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8568 - val_loss: 0.3752 - val_accuracy: 0.8531 Epoch 25/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3610 - accuracy: 0.8512 - val_loss: 0.3632 - val_accuracy: 0.8469 Epoch 26/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3588 - accuracy: 0.8506 - val_loss: 0.3640 - val_accuracy: 0.8469 Epoch 27/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3621 - accuracy: 0.8518 - val_loss: 0.3699 - val_accuracy: 0.8547 Epoch 28/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3605 - accuracy: 0.8533 - val_loss: 0.3709 - val_accuracy: 0.8508 Epoch 29/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3598 - accuracy: 0.8523 - val_loss: 0.3677 - val_accuracy: 0.8539 Epoch 30/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3502 - accuracy: 0.8547 - val_loss: 0.3634 - val_accuracy: 0.8516 Epoch 31/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3530 - accuracy: 0.8504 - val_loss: 0.3743 - val_accuracy: 0.8562 Epoch 32/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3511 - accuracy: 0.8535 - val_loss: 0.3610 - val_accuracy: 0.8531 Epoch 33/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3483 - accuracy: 0.8541 - val_loss: 0.3994 - val_accuracy: 0.8562 Epoch 34/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3538 - accuracy: 0.8559 - val_loss: 0.3675 - val_accuracy: 0.8539 Epoch 35/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3602 - accuracy: 0.8541 - val_loss: 0.3685 - val_accuracy: 0.8516 Epoch 36/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3594 - accuracy: 0.8500 - val_loss: 0.3665 - val_accuracy: 0.8516 Epoch 37/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3517 - accuracy: 0.8574 - val_loss: 0.3620 - val_accuracy: 0.8555 Epoch 38/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3516 - accuracy: 0.8539 - val_loss: 0.3639 - val_accuracy: 0.8555 Epoch 39/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3500 - accuracy: 0.8531 - val_loss: 0.3656 - val_accuracy: 0.8461 Epoch 40/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3564 - accuracy: 0.8551 - val_loss: 0.3642 - val_accuracy: 0.8508 Epoch 41/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3491 - accuracy: 0.8551 - val_loss: 0.3615 - val_accuracy: 0.8547 Epoch 42/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8529 - val_loss: 0.3666 - val_accuracy: 0.8523 Epoch 43/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3562 - accuracy: 0.8516 - val_loss: 0.3716 - val_accuracy: 0.8508 Epoch 44/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3488 - accuracy: 0.8572 - val_loss: 0.3554 - val_accuracy: 0.8578 Epoch 45/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3584 - accuracy: 0.8562 - val_loss: 0.3741 - val_accuracy: 0.8516 Epoch 46/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3482 - accuracy: 0.8592 - val_loss: 0.3619 - val_accuracy: 0.8477 Epoch 47/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3555 - accuracy: 0.8527 - val_loss: 0.3627 - val_accuracy: 0.8469 Epoch 48/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3470 - accuracy: 0.8590 - val_loss: 0.3644 - val_accuracy: 0.8500 Epoch 49/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3484 - accuracy: 0.8578 - val_loss: 0.3621 - val_accuracy: 0.8484 Epoch 50/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3508 - accuracy: 0.8539 - val_loss: 0.3562 - val_accuracy: 0.8547
#Plotting Train Loss vs Validation Loss
plt.plot(history_5.history['loss'])
plt.plot(history_5.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
from sklearn.metrics import roc_curve
from matplotlib import pyplot
# predict probabilities
yhat5 = estimator_v5.predict(X_test)
# keep probabilities for the positive outcome only
yhat5 = yhat5[:, 0]
# calculate roc curves
fpr, tpr, thresholds5 = roc_curve(y_test, yhat5)
# calculate the g-mean for each threshold
gmeans5 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans5)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds5[ix], gmeans5[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Churn')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()
Best Threshold=0.204202, G-Mean=0.779
y_pred_e5=estimator_v5.predict(X_test)
y_pred_e5 = (y_pred_e5 > thresholds5[ix])
y_pred_e5
array([[False],
[False],
[False],
...,
[False],
[False],
[ True]])
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm5=confusion_matrix(y_test, y_pred_e5)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm5,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr5=metrics.classification_report(y_test,y_pred_e5)
print(cr5)
precision recall f1-score support
0.0 0.93 0.80 0.86 1585
1.0 0.49 0.76 0.60 415
accuracy 0.79 2000
macro avg 0.71 0.78 0.73 2000
weighted avg 0.84 0.79 0.80 2000
from tensorflow import keras
from tensorflow.keras import layers
from keras_tuner.tuners import RandomSearch
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
def build_model(h):
model = keras.Sequential()
for i in range(h.Int('num_layers', 2, 10)):
model.add(layers.Dense(units=h.Int('units_' + str(i),
min_value=32,
max_value=256,
step=32),
activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(
optimizer=keras.optimizers.Adam(
h.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
loss='binary_crossentropy',
metrics=['accuracy'])
return model
tuner = RandomSearch(
build_model,
objective='val_accuracy',
max_trials=5,
executions_per_trial=3,
project_name='Job_')
tuner.search_space_summary()
Search space summary
Default search space size: 4
num_layers (Int)
{'default': None, 'conditions': [], 'min_value': 2, 'max_value': 10, 'step': 1, 'sampling': None}
units_0 (Int)
{'default': None, 'conditions': [], 'min_value': 32, 'max_value': 256, 'step': 32, 'sampling': None}
units_1 (Int)
{'default': None, 'conditions': [], 'min_value': 32, 'max_value': 256, 'step': 32, 'sampling': None}
learning_rate (Choice)
{'default': 0.01, 'conditions': [], 'values': [0.01, 0.001, 0.0001], 'ordered': True}
### Searching the best model on X and y train
tuner.search(X_train, y_train,
epochs=5,
validation_split = 0.2)
Trial 5 Complete [00h 00m 07s] val_accuracy: 0.8158854047457377 Best val_accuracy So Far: 0.8565104206403097 Total elapsed time: 00h 00m 33s INFO:tensorflow:Oracle triggered exit
## Printing the best models with their hyperparameters
tuner.results_summary()
Results summary Results in ./Job_ Showing 10 best trials <keras_tuner.engine.objective.Objective object at 0x7fdb19aa7ac0> Trial summary Hyperparameters: num_layers: 5 units_0: 64 units_1: 96 learning_rate: 0.001 units_2: 96 units_3: 224 units_4: 256 units_5: 160 units_6: 192 units_7: 224 units_8: 224 Score: 0.8565104206403097 Trial summary Hyperparameters: num_layers: 5 units_0: 160 units_1: 160 learning_rate: 0.001 units_2: 224 units_3: 128 units_4: 224 units_5: 64 units_6: 160 units_7: 64 units_8: 32 Score: 0.854687511920929 Trial summary Hyperparameters: num_layers: 8 units_0: 160 units_1: 64 learning_rate: 0.001 units_2: 160 units_3: 64 units_4: 192 units_5: 32 units_6: 224 units_7: 96 units_8: 192 Score: 0.854687492052714 Trial summary Hyperparameters: num_layers: 9 units_0: 128 units_1: 32 learning_rate: 0.0001 units_2: 160 units_3: 160 units_4: 160 units_5: 192 units_6: 96 units_7: 128 units_8: 96 Score: 0.8158854047457377 Trial summary Hyperparameters: num_layers: 9 units_0: 192 units_1: 96 learning_rate: 0.0001 units_2: 32 units_3: 32 units_4: 32 units_5: 32 units_6: 32 units_7: 32 units_8: 32 Score: 0.7911458214124044
0.8565104206403097 best accuracy so far with 5 layers
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
model7 = Sequential()
model7.add(Dense(64,activation='relu',kernel_initializer='he_uniform',input_dim = X_train.shape[1]))
model7.add(Dense(96,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(96,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(224,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(256,activation='relu',kernel_initializer='he_uniform'))
model7.add(Dense(1, activation = 'sigmoid'))
model7.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 64) 832
dense_1 (Dense) (None, 96) 6240
dense_2 (Dense) (None, 96) 9312
dense_3 (Dense) (None, 224) 21728
dense_4 (Dense) (None, 256) 57600
dense_5 (Dense) (None, 1) 257
=================================================================
Total params: 95,969
Trainable params: 95,969
Non-trainable params: 0
_________________________________________________________________
optimizer = tf.keras.optimizers.Adam(0.001)
model7.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=['accuracy'])
history_7 = model7.fit(X_train,y_train,batch_size=64,epochs=50,verbose=1,validation_split = 0.2)
Epoch 1/50 80/80 [==============================] - 1s 3ms/step - loss: 0.4649 - accuracy: 0.8021 - val_loss: 0.4701 - val_accuracy: 0.8062 Epoch 2/50 80/80 [==============================] - 0s 2ms/step - loss: 0.4301 - accuracy: 0.8156 - val_loss: 0.4187 - val_accuracy: 0.8227 Epoch 3/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3922 - accuracy: 0.8346 - val_loss: 0.4038 - val_accuracy: 0.8414 Epoch 4/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3642 - accuracy: 0.8453 - val_loss: 0.3926 - val_accuracy: 0.8430 Epoch 5/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3474 - accuracy: 0.8553 - val_loss: 0.3756 - val_accuracy: 0.8484 Epoch 6/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3462 - accuracy: 0.8543 - val_loss: 0.3914 - val_accuracy: 0.8336 Epoch 7/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3296 - accuracy: 0.8641 - val_loss: 0.3877 - val_accuracy: 0.8375 Epoch 8/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3263 - accuracy: 0.8641 - val_loss: 0.3891 - val_accuracy: 0.8352 Epoch 9/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3192 - accuracy: 0.8668 - val_loss: 0.3833 - val_accuracy: 0.8352 Epoch 10/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3104 - accuracy: 0.8711 - val_loss: 0.4173 - val_accuracy: 0.8344 Epoch 11/50 80/80 [==============================] - 0s 2ms/step - loss: 0.3063 - accuracy: 0.8687 - val_loss: 0.3938 - val_accuracy: 0.8508 Epoch 12/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2974 - accuracy: 0.8701 - val_loss: 0.3895 - val_accuracy: 0.8367 Epoch 13/50 80/80 [==============================] - 0s 3ms/step - loss: 0.2927 - accuracy: 0.8797 - val_loss: 0.3951 - val_accuracy: 0.8367 Epoch 14/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2877 - accuracy: 0.8803 - val_loss: 0.4095 - val_accuracy: 0.8383 Epoch 15/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2748 - accuracy: 0.8846 - val_loss: 0.4051 - val_accuracy: 0.8359 Epoch 16/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2711 - accuracy: 0.8844 - val_loss: 0.4207 - val_accuracy: 0.8195 Epoch 17/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2687 - accuracy: 0.8844 - val_loss: 0.4224 - val_accuracy: 0.8258 Epoch 18/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2587 - accuracy: 0.8953 - val_loss: 0.4372 - val_accuracy: 0.8492 Epoch 19/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2552 - accuracy: 0.8910 - val_loss: 0.4543 - val_accuracy: 0.8422 Epoch 20/50 80/80 [==============================] - 0s 3ms/step - loss: 0.2479 - accuracy: 0.8934 - val_loss: 0.4422 - val_accuracy: 0.8438 Epoch 21/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2403 - accuracy: 0.8990 - val_loss: 0.4658 - val_accuracy: 0.8281 Epoch 22/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2306 - accuracy: 0.9021 - val_loss: 0.4546 - val_accuracy: 0.8336 Epoch 23/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2197 - accuracy: 0.9059 - val_loss: 0.4966 - val_accuracy: 0.8352 Epoch 24/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2194 - accuracy: 0.9029 - val_loss: 0.5221 - val_accuracy: 0.8406 Epoch 25/50 80/80 [==============================] - 0s 2ms/step - loss: 0.2094 - accuracy: 0.9113 - val_loss: 0.5080 - val_accuracy: 0.8070 Epoch 26/50 80/80 [==============================] - 0s 3ms/step - loss: 0.2048 - accuracy: 0.9094 - val_loss: 0.5256 - val_accuracy: 0.8203 Epoch 27/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1950 - accuracy: 0.9176 - val_loss: 0.5459 - val_accuracy: 0.8117 Epoch 28/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1863 - accuracy: 0.9195 - val_loss: 0.5628 - val_accuracy: 0.8242 Epoch 29/50 80/80 [==============================] - 0s 3ms/step - loss: 0.1698 - accuracy: 0.9283 - val_loss: 0.5982 - val_accuracy: 0.8180 Epoch 30/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1872 - accuracy: 0.9238 - val_loss: 0.5737 - val_accuracy: 0.8305 Epoch 31/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1754 - accuracy: 0.9252 - val_loss: 0.6082 - val_accuracy: 0.8297 Epoch 32/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1580 - accuracy: 0.9318 - val_loss: 0.6193 - val_accuracy: 0.8289 Epoch 33/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1437 - accuracy: 0.9377 - val_loss: 0.7511 - val_accuracy: 0.8289 Epoch 34/50 80/80 [==============================] - 0s 3ms/step - loss: 0.1463 - accuracy: 0.9373 - val_loss: 0.7235 - val_accuracy: 0.8336 Epoch 35/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1563 - accuracy: 0.9354 - val_loss: 0.6715 - val_accuracy: 0.7883 Epoch 36/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1509 - accuracy: 0.9381 - val_loss: 0.7063 - val_accuracy: 0.8047 Epoch 37/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1446 - accuracy: 0.9395 - val_loss: 0.7154 - val_accuracy: 0.8320 Epoch 38/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1438 - accuracy: 0.9387 - val_loss: 0.7652 - val_accuracy: 0.8211 Epoch 39/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1223 - accuracy: 0.9488 - val_loss: 0.7570 - val_accuracy: 0.8180 Epoch 40/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1124 - accuracy: 0.9531 - val_loss: 0.7971 - val_accuracy: 0.8211 Epoch 41/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1098 - accuracy: 0.9543 - val_loss: 0.9006 - val_accuracy: 0.8258 Epoch 42/50 80/80 [==============================] - 0s 3ms/step - loss: 0.1050 - accuracy: 0.9594 - val_loss: 0.8420 - val_accuracy: 0.8188 Epoch 43/50 80/80 [==============================] - 0s 2ms/step - loss: 0.0929 - accuracy: 0.9625 - val_loss: 0.9249 - val_accuracy: 0.8070 Epoch 44/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1017 - accuracy: 0.9604 - val_loss: 0.9175 - val_accuracy: 0.8016 Epoch 45/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1107 - accuracy: 0.9572 - val_loss: 0.8992 - val_accuracy: 0.8141 Epoch 46/50 80/80 [==============================] - 0s 2ms/step - loss: 0.1159 - accuracy: 0.9535 - val_loss: 0.9143 - val_accuracy: 0.8133 Epoch 47/50 80/80 [==============================] - 0s 2ms/step - loss: 0.0827 - accuracy: 0.9662 - val_loss: 0.9381 - val_accuracy: 0.8141 Epoch 48/50 80/80 [==============================] - 0s 2ms/step - loss: 0.0787 - accuracy: 0.9695 - val_loss: 1.0746 - val_accuracy: 0.8062 Epoch 49/50 80/80 [==============================] - 0s 2ms/step - loss: 0.0852 - accuracy: 0.9680 - val_loss: 1.0367 - val_accuracy: 0.8172 Epoch 50/50 80/80 [==============================] - 0s 2ms/step - loss: 0.0779 - accuracy: 0.9695 - val_loss: 1.0493 - val_accuracy: 0.8117
#Plotting Train Loss vs Validation Loss
plt.plot(history_7.history['loss'])
plt.plot(history_7.history['val_loss'])
plt.title('model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
from sklearn.metrics import roc_curve
from matplotlib import pyplot
# predict probabilities
yhat7 = model7.predict(X_test)
# keep probabilities for the positive outcome only
yhat7 = yhat7[:, 0]
# calculate roc curves
fpr, tpr, thresholds7 = roc_curve(y_test, yhat7)
# calculate the g-mean for each threshold
gmeans7 = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans7)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds7[ix], gmeans7[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()
Best Threshold=0.023485, G-Mean=0.717
y_pred_e7=model7.predict(X_test)
y_pred_e7 = (y_pred_e7 > thresholds7[ix])
y_pred_e7
array([[False],
[ True],
[False],
...,
[False],
[False],
[ True]])
#Calculating the confusion matrix
from sklearn.metrics import confusion_matrix
cm7=confusion_matrix(y_test, y_pred_e7)
labels = ['True Positive','False Negative','False Positive','True Negative']
categories = [ 'Not Churning','Churning']
make_confusion_matrix(cm7,
group_names=labels,
categories=categories,
cmap='Blues')
#Accuracy as per the classification report
from sklearn import metrics
cr7=metrics.classification_report(y_test,y_pred_e7)
print(cr7)
precision recall f1-score support
0.0 0.91 0.71 0.80 1585
1.0 0.39 0.72 0.51 415
accuracy 0.71 2000
macro avg 0.65 0.72 0.65 2000
weighted avg 0.80 0.71 0.74 2000
print(cr3) # with dropout
print(cr4)# Random Search CV
print(cr5)# Grid Search CV
print(cr7) #Keras Tuner
precision recall f1-score support
0.0 0.92 0.83 0.87 1585
1.0 0.52 0.73 0.61 415
accuracy 0.80 2000
macro avg 0.72 0.78 0.74 2000
weighted avg 0.84 0.80 0.82 2000
precision recall f1-score support
0.0 0.93 0.80 0.86 1585
1.0 0.49 0.76 0.60 415
accuracy 0.79 2000
macro avg 0.71 0.78 0.73 2000
weighted avg 0.84 0.79 0.80 2000
precision recall f1-score support
0.0 0.93 0.80 0.86 1585
1.0 0.49 0.76 0.60 415
accuracy 0.79 2000
macro avg 0.71 0.78 0.73 2000
weighted avg 0.84 0.79 0.80 2000
precision recall f1-score support
0.0 0.91 0.71 0.80 1585
1.0 0.39 0.72 0.51 415
accuracy 0.71 2000
macro avg 0.65 0.72 0.65 2000
weighted avg 0.80 0.71 0.74 2000
Our model here can be Model 4 which uses the Dropout regularization technique and works on the imbalanced dataset.
GridSearchCV, RandomSearch CV has better ROC curves model's F1 score, while better than in Randomized Search, is slightly lower than in Model 4 (the Dropout model) which can also be used.
Any one Machine Learning model above can be used.
We cab do better feature engineerning by removing the flaws of the skewed variables if required.
Based on the EDA, we can target the churning categories like older age group and based on other demography like country Germany etc as well. The model will learn better as data gets less imbalanced. Existing model gives excellent Acurracy and F1 score
We encourage you to further look to optimize the model and come up with better results.
Thank you - Amogh